update docs

Author: vpj

docs/diffusion/ddpm/unet.html

@@ -1017,7 +1017,7 @@ <h2>U-Net</h2>
         <div class='code'>
             <div class="highlight"><pre><span class="lineno">313</span>    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">image_channels</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">3</span><span class="p">,</span> <span class="n">n_channels</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">64</span><span class="p">,</span>
 <span class="lineno">314</span>                 <span class="n">ch_mults</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">int</span><span class="p">,</span> <span class="o">...</span><span class="p">],</span> <span class="n">List</span><span class="p">[</span><span class="nb">int</span><span class="p">]]</span> <span class="o">=</span> <span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">4</span><span class="p">),</span>
-<span class="lineno">315</span>                 <span class="n">is_attn</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">bool</span><span class="p">,</span> <span class="o">...</span><span class="p">],</span> <span class="n">List</span><span class="p">[</span><span class="nb">int</span><span class="p">]]</span> <span class="o">=</span> <span class="p">(</span><span class="kc">False</span><span class="p">,</span> <span class="kc">False</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="kc">True</span><span class="p">),</span>
+<span class="lineno">315</span>                 <span class="n">is_attn</span><span class="p">:</span> <span class="n">Union</span><span class="p">[</span><span class="n">Tuple</span><span class="p">[</span><span class="nb">bool</span><span class="p">,</span> <span class="o">...</span><span class="p">],</span> <span class="n">List</span><span class="p">[</span><span class="nb">bool</span><span class="p">]]</span> <span class="o">=</span> <span class="p">(</span><span class="kc">False</span><span class="p">,</span> <span class="kc">False</span><span class="p">,</span> <span class="kc">True</span><span class="p">,</span> <span class="kc">True</span><span class="p">),</span>
 <span class="lineno">316</span>                 <span class="n">n_blocks</span><span class="p">:</span> <span class="nb">int</span> <span class="o">=</span> <span class="mi">2</span><span class="p">):</span></pre></div>
         </div>
     </div>

docs/normalization/group_norm/readme.html

@@ -75,7 +75,7 @@ <h1><a href="https://nn.labml.ai/normalization/group_norm/index.html">Group Norm
 <p>This is a <a href="https://pytorch.org">PyTorch</a> implementation of the <a href="https://arxiv.org/abs/1803.08494">Group Normalization</a> paper.</p>
 <p><a href="https://nn.labml.ai/normalization/batch_norm/index.html">Batch Normalization</a> works well for large enough batch sizes but not well for small batch sizes, because it normalizes over the batch. Training large models with large batch sizes is not possible due to the memory capacity of the devices.</p>
 <p>This paper introduces Group Normalization, which normalizes a set of features together as a group. This is based on the observation that classical features such as <a href="https://en.wikipedia.org/wiki/Scale-invariant_feature_transform">SIFT</a> and <a href="https://en.wikipedia.org/wiki/Histogram_of_oriented_gradients">HOG</a> are group-wise features. The paper proposes dividing feature channels into groups and then separately normalizing all channels within each group.</p>
-<p>Here&#x27;s a <a href="https://nn.labml.ai/normalization/group_norm/experiment.html">CIFAR 10 classification model</a> that uses instance normalization.</p>
+<p>Here&#x27;s a <a href="https://nn.labml.ai/normalization/group_norm/experiment.html">CIFAR 10 classification model</a> that uses group normalization.</p>
 <p><a href="https://colab.research.google.com/github/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/normalization/group_norm/experiment.ipynb"><img alt="Open In Colab" src="https://colab.research.google.com/assets/colab-badge.svg"></a> </p>
 
         </div>

docs/sitemap.xml

@@ -645,7 +645,7 @@
 
     <url>
       <loc>https://nn.labml.ai/diffusion/ddpm/unet.html</loc>
-      <lastmod>2023-02-17T16:30:00+00:00</lastmod>
+      <lastmod>2023-11-06T16:30:00+00:00</lastmod>
       <priority>1.00</priority>
     </url>
 
@@ -1072,7 +1072,7 @@
 
     <url>
       <loc>https://nn.labml.ai/transformers/vit/index.html</loc>
-      <lastmod>2023-10-24T16:30:00+00:00</lastmod>
+      <lastmod>2023-11-06T16:30:00+00:00</lastmod>
       <priority>1.00</priority>
     </url>
 

docs/transformers/vit/index.html

@@ -570,25 +570,25 @@ <h2>Vision Transformer</h2>
             <div class='section-link'>
                 <a href='#section-37'>#</a>
             </div>
-            <p>Add positional embeddings </p>
+            <p>Concatenate the <code  class="highlight"><span></span><span class="p">[</span><span class="n">CLS</span><span class="p">]</span></code>
+ token embeddings before feeding the transformer </p>
 
         </div>
         <div class='code'>
-            <div class="highlight"><pre><span class="lineno">195</span>        <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">pos_emb</span><span class="p">(</span><span class="n">x</span><span class="p">)</span></pre></div>
+            <div class="highlight"><pre><span class="lineno">195</span>        <span class="n">cls_token_emb</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">cls_token_emb</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
+<span class="lineno">196</span>        <span class="n">x</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">cat</span><span class="p">([</span><span class="n">cls_token_emb</span><span class="p">,</span> <span class="n">x</span><span class="p">])</span></pre></div>
         </div>
     </div>
     <div class='section' id='section-38'>
         <div class='docs'>
             <div class='section-link'>
                 <a href='#section-38'>#</a>
             </div>
-            <p>Concatenate the <code  class="highlight"><span></span><span class="p">[</span><span class="n">CLS</span><span class="p">]</span></code>
- token embeddings before feeding the transformer </p>
+            <p>Add positional embeddings </p>
 
         </div>
         <div class='code'>
-            <div class="highlight"><pre><span class="lineno">197</span>        <span class="n">cls_token_emb</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">cls_token_emb</span><span class="o">.</span><span class="n">expand</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span> <span class="n">x</span><span class="o">.</span><span class="n">shape</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
-<span class="lineno">198</span>        <span class="n">x</span> <span class="o">=</span> <span class="n">torch</span><span class="o">.</span><span class="n">cat</span><span class="p">([</span><span class="n">cls_token_emb</span><span class="p">,</span> <span class="n">x</span><span class="p">])</span></pre></div>
+            <div class="highlight"><pre><span class="lineno">198</span>        <span class="n">x</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">pos_emb</span><span class="p">(</span><span class="n">x</span><span class="p">)</span></pre></div>
         </div>
     </div>
     <div class='section' id='section-39'>