YSMull
<-- home

Feed

YSMull https://ysmull.cn 减少无谓的事务创建 <ul id="markdown-toc"> <li><a href="#id-问题背景" id="markdown-toc-id-问题背景">问题背景</a></li> <li><a href="#id-降低-io-规模" id="markdown-toc-id-降低-io-规模">降低 IO 规模</a></li> <li><a href="#id-小结" id="markdown-toc-id-小结">小结</a></li> </ul> <h2 id="id-问题背景">问题背景</h2> <p>今天要修复某业务列表加载缓慢的问题,业务逻辑的写法是在双重循环的最内层循环中有数据库查询,这块代码当初被写成这样,是因为业务本身就是这么复杂导致的,大多数场景不会遇到性能瓶颈,但当客户的业务量大到一定程度后,用户体验和服务的性能都会收到影响。<br /> 伪代码如下:</p> <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for</span> <span class="p">(</span><span class="kd">let</span> <span class="nx">projectId</span> <span class="k">of</span> <span class="nb">Object</span><span class="p">.</span><span class="nx">keys</span><span class="p">(</span><span class="nx">xxxMap</span><span class="p">))</span> <span class="p">{</span> <span class="kd">let</span> <span class="nx">projectItems</span> <span class="o">=</span> <span class="nx">xxxMap</span><span class="p">[</span><span class="nx">projectId</span><span class="p">];</span> <span class="k">for</span> <span class="p">(</span><span class="kd">let</span> <span class="nx">item</span> <span class="k">of</span> <span class="nx">projectItems</span><span class="p">)</span> <span class="p">{</span> <span class="kd">let</span> <span class="nx">users</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">getUsersByReceivers</span><span class="p">(</span><span class="nx">item</span><span class="p">.</span><span class="nx">receiverList</span><span class="p">,</span> <span class="nx">projectId</span><span class="p">);</span> <span class="c1">// 数据库 IO 操作</span> <span class="k">if</span> <span class="p">(</span><span class="nx">check</span><span class="p">(</span><span class="nx">users</span><span class="p">))</span> <span class="p">{</span> <span class="c1">// ...</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> </code></pre></div></div> <p>双重循环的串行 IO 是非常耗时的操作,我们先不讨论其业务逻辑,对上面这段代码进行一定的变换,提前就并发地把内层循环需要使用到的数据查询好。</p> <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for</span> <span class="p">(</span><span class="kd">let</span> <span class="nx">projectId</span> <span class="k">of</span> <span class="nb">Object</span><span class="p">.</span><span class="nx">keys</span><span class="p">(</span><span class="nx">xxxMap</span><span class="p">))</span> <span class="p">{</span> <span class="kd">let</span> <span class="nx">projectItems</span> <span class="o">=</span> <span class="nx">xxxMap</span><span class="p">[</span><span class="nx">projectId</span><span class="p">];</span> <span class="kd">let</span> <span class="nx">usersList</span> <span class="o">=</span> <span class="k">await</span> <span class="nb">Promise</span><span class="p">.</span><span class="nx">all</span><span class="p">(</span><span class="nx">projectItems</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="k">async</span> <span class="p">(</span><span class="nx">item</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span> <span class="k">return</span> <span class="k">await</span> <span class="nx">getUsersByReceivers</span><span class="p">(</span><span class="nx">item</span><span class="p">.</span><span class="nx">receiverList</span><span class="p">,</span> <span class="nx">projectId</span><span class="p">);</span> <span class="p">}));</span> <span class="kd">let</span> <span class="nx">usersListMap</span> <span class="o">=</span> <span class="nx">_</span><span class="p">.</span><span class="nx">zipObject</span><span class="p">(</span><span class="nx">projectItems</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="nx">c</span> <span class="o">=&gt;</span> <span class="nx">c</span><span class="p">.</span><span class="nx">id</span><span class="p">),</span> <span class="nx">usersList</span><span class="p">);</span> <span class="k">for</span> <span class="p">(</span><span class="kd">let</span> <span class="nx">item</span> <span class="k">of</span> <span class="nx">projectItems</span><span class="p">)</span> <span class="p">{</span> <span class="k">if</span> <span class="p">(</span><span class="nx">check</span><span class="p">(</span><span class="nx">usersListMap</span><span class="p">[</span><span class="nx">item</span><span class="p">.</span><span class="nx">id</span><span class="p">]))</span> <span class="p">{</span> <span class="c1">// ...</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> </code></pre></div></div> <p>这样修改后,我本地使用测试账号对目标接口进行测试,耗时从 140ms 左右降低到了 50ms 左右,此时我认为已经满足了客户的需求。</p> <p>提交代码后,同事在进行 code review 时提出,这样虽然提高了性能,但可能会对数据库产生更大的压力,并没有减少产生的 SQL 语句个数,应该从业务逻辑上另外想办法绕过,不要让 IO 规模跟 <code class="language-plaintext highlighter-rouge">projectItems</code> 的规模产生关系。</p> <h2 id="id-降低-io-规模">降低 IO 规模</h2> <p>上面的业务代码产生的数据库 IO,不仅跟项目个数有关,也跟每个项目下面的业务项的个数有关,重新梳理后,可以让 IO 次数只与项目个数有关,但测试后发现,性能衰减到了 70ms 左右。数了数发起的 SQL 数目,从之前的 14 个 SQL 减少到了 9 个 SQL。一顿操作后反而性能有所下降,这让我比较困惑。</p> <p>查看 SQL 日志发现,有大量的事务创建和提交,怀疑是此带来的额外开销。</p> <pre><code class="language-log">[ayH2ah] [2021-06-28 19:49:11.49.272] beginTransaction [ayH2ah] [2021-06-28 19:49:11.49.287] select role.* from role where ( role.id in (2731, 2725) ) order [ayH2ah] [2021-06-28 19:49:11.49.290] select xxx.* from xxx where ( xxx.id in (338) ) order by create_ [ayH2ah] [2021-06-28 19:49:11.49.291] commitTransaction [oJLwYA] [2021-06-28 19:49:11.49.298] select xxx.*, uprv.role_id as roleId from (select * from (select [hLKZvf] [2021-06-28 19:49:11.49.301] beginTransaction [hLKZvf] [2021-06-28 19:49:11.49.304] select role.* from role where ( role.id in (1) ) order by create [hLKZvf] [2021-06-28 19:49:11.49.306] select xxx.* from xxx where ( xxx.id in (338) ) order by create_ [hLKZvf] [2021-06-28 19:49:11.49.307] commitTransaction [69WZw4] [2021-06-28 19:49:11.49.314] select xxx.*, uprv.role_id as roleId from (select * from (select [nvwxW7] [2021-06-28 19:49:11.49.317] beginTransaction [nvwxW7] [2021-06-28 19:49:11.49.320] select xxx.* from xxx where ( xxx.id in (338) ) order by create_ [nvwxW7] [2021-06-28 19:49:11.49.321] commitTransaction [vLUcHc] [2021-06-28 19:49:11.49.324] beginTransaction [vLUcHc] [2021-06-28 19:49:11.49.327] select xxx.* from xxx where ( xxx.id in (338) ) order by create_ [vLUcHc] [2021-06-28 19:49:11.49.328] commitTransaction [dNKXAN] [2021-06-28 19:49:11.49.331] beginTransaction [dNKXAN] [2021-06-28 19:49:11.49.334] select xxx.* from xxx where ( xxx.id in (338) ) order by create_ [dNKXAN] [2021-06-28 19:49:11.49.335] commitTransaction </code></pre> <p>将这些业务放在同一个事务中之后,性能恢复正常,回到了 50ms,此时代码变成了</p> <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">for</span> <span class="p">(</span><span class="kd">let</span> <span class="nx">projectId</span> <span class="k">of</span> <span class="nb">Object</span><span class="p">.</span><span class="nx">keys</span><span class="p">(</span><span class="nx">xxxMap</span><span class="p">))</span> <span class="p">{</span> <span class="kd">let</span> <span class="nx">projectItems</span> <span class="o">=</span> <span class="nx">xxxMap</span><span class="p">[</span><span class="nx">projectId</span><span class="p">];</span> <span class="kd">let</span> <span class="nx">usersListMap</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">getUsersMapByReceivers</span><span class="p">(</span><span class="nx">projectItems</span><span class="p">,</span> <span class="nx">projectId</span><span class="p">);</span> <span class="c1">// 该方法产生的 IO 数与 projectItems 规模无关</span> <span class="k">for</span> <span class="p">(</span><span class="kd">let</span> <span class="nx">item</span> <span class="k">of</span> <span class="nx">projectItems</span><span class="p">)</span> <span class="p">{</span> <span class="k">if</span> <span class="p">(</span><span class="nx">check</span><span class="p">(</span><span class="nx">usersListMap</span><span class="p">[</span><span class="nx">item</span><span class="p">.</span><span class="nx">id</span><span class="p">]))</span> <span class="p">{</span> <span class="c1">// ...</span> <span class="p">}</span> <span class="p">}</span> <span class="p">}</span> </code></pre></div></div> <p>此时还可以使用一开始的并发 IO 的方式继续优化,彻底去掉 for 循环产生的串行 IO,性能来到 35ms,代码略。</p> <p>碎片事务过多导致性能降低的一个原因似乎是,node-mysql 在创建事务和提交事务时,会向数据库提交独立的 SQL 语句:</p> <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">Connection</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">beginTransaction</span> <span class="o">=</span> <span class="kd">function</span> <span class="nx">beginTransaction</span><span class="p">(</span><span class="nx">options</span><span class="p">,</span> <span class="nx">callback</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// ....</span> <span class="nx">options</span> <span class="o">=</span> <span class="nx">options</span> <span class="o">||</span> <span class="p">{};</span> <span class="nx">options</span><span class="p">.</span><span class="nx">sql</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">START TRANSACTION</span><span class="dl">'</span><span class="p">;</span> <span class="nx">options</span><span class="p">.</span><span class="nx">values</span> <span class="o">=</span> <span class="kc">null</span><span class="p">;</span> <span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">query</span><span class="p">(</span><span class="nx">options</span><span class="p">,</span> <span class="nx">callback</span><span class="p">);</span> <span class="p">};</span> <span class="nx">Connection</span><span class="p">.</span><span class="nx">prototype</span><span class="p">.</span><span class="nx">commit</span> <span class="o">=</span> <span class="kd">function</span> <span class="nx">commit</span><span class="p">(</span><span class="nx">options</span><span class="p">,</span> <span class="nx">callback</span><span class="p">)</span> <span class="p">{</span> <span class="c1">// ...</span> <span class="nx">options</span> <span class="o">=</span> <span class="nx">options</span> <span class="o">||</span> <span class="p">{};</span> <span class="nx">options</span><span class="p">.</span><span class="nx">sql</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">COMMIT</span><span class="dl">'</span><span class="p">;</span> <span class="nx">options</span><span class="p">.</span><span class="nx">values</span> <span class="o">=</span> <span class="kc">null</span><span class="p">;</span> <span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">query</span><span class="p">(</span><span class="nx">options</span><span class="p">,</span> <span class="nx">callback</span><span class="p">);</span> <span class="p">};</span> </code></pre></div></div> <h2 id="id-小结">小结</h2> <p>优化业务代码的性能,不仅可以从 SQL 本身出发去优化单个 SQL 的性能,也可以从业务逻辑上考虑。<br /> 具体地说,我们需要尽可能的避免在循环中去发起数据库查询,另外要注意不要产生过多的碎片事务。</p> 2021-06-28T00:00:00+00:00 https://ysmull.cn/blog/reduce-trans.html https://ysmull.cn/blog/reduce-trans.html 如何处理 Nodejs 服务初始化配置中的异步 IO <ul id="markdown-toc"> <li><a href="#id-背景" id="markdown-toc-id-背景">背景</a></li> <li><a href="#id-问题解决" id="markdown-toc-id-问题解决">问题解决</a> <ul> <li><a href="#id-双进程法" id="markdown-toc-id-双进程法">双进程法</a></li> <li><a href="#id-双生命周期法" id="markdown-toc-id-双生命周期法">双生命周期法</a></li> </ul> </li> </ul> <h2 id="id-背景">背景</h2> <p>在 Nodejs 服务中,我们往往会引入一个 Config 模块,该模块是整个 App 的配置中心,其加载将会初始化好整个系统所需的配置。<br /> 例如:</p> <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// config.js</span> <span class="kd">let</span> <span class="nx">config</span> <span class="o">=</span> <span class="p">{};</span> <span class="kd">let</span> <span class="nx">init</span> <span class="o">=</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span> <span class="nx">config</span> <span class="o">=</span> <span class="nx">initConfig</span><span class="p">();</span> <span class="nx">process</span><span class="p">.</span><span class="nx">globalConfig</span> <span class="o">=</span> <span class="nx">config</span><span class="p">;</span> <span class="p">}</span> <span class="nx">init</span><span class="p">();</span> <span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span> <span class="na">get</span><span class="p">:</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">key</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="nx">config</span><span class="p">[</span><span class="nx">key</span><span class="p">];</span> <span class="p">}</span> <span class="p">}</span> </code></pre></div></div> <p>在上面的代码中,init方法做了两件事情,分别满足了两种使用场景:</p> <ol> <li>先通过调用 initConfig() 方法构建了 config 对象,这样其它模块可以引入该模块后通过暴露出去的 get 方法获取配置。 <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="kd">const</span> <span class="nx">config</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">./config.js</span><span class="dl">'</span><span class="p">);</span> <span class="nx">exports</span><span class="p">.</span><span class="nx">serviceA</span> <span class="o">=</span> <span class="k">async</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span> <span class="kd">let</span> <span class="nx">configA</span> <span class="o">=</span> <span class="nx">config</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">'</span><span class="s1">keyA</span><span class="dl">'</span><span class="p">);</span> <span class="c1">// ...</span> <span class="p">}</span> </code></pre></div> </div> </li> <li>把 config 对象绑定到 process 这个全局对象上,这样其它模块可以通过 process.globalConfig 获取到 config 对象。 比如某些模块,可能需要基于 process.globalConfig 来进行初始化: <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="c1">// other module</span> <span class="o">/</span> <span class="p">...</span> <span class="nx">exports</span><span class="p">.</span><span class="nx">obj</span> <span class="o">=</span> <span class="nx">initObj</span><span class="p">(</span><span class="nx">process</span><span class="p">.</span><span class="nx">globalConfig</span><span class="p">);</span> <span class="c1">// ...</span> </code></pre></div> </div> </li> </ol> <p>如果我们在的 initConfig() 需要发一个请求获取配置,那么 config.js 就得改写为:</p> <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// config.js</span> <span class="kd">let</span> <span class="nx">config</span> <span class="o">=</span> <span class="p">{};</span> <span class="kd">let</span> <span class="nx">init</span> <span class="o">=</span> <span class="k">async</span> <span class="kd">function</span> <span class="p">()</span> <span class="p">{</span> <span class="nx">config</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">initConfig</span><span class="p">();</span> <span class="nx">process</span><span class="p">.</span><span class="nx">globalConfig</span> <span class="o">=</span> <span class="nx">config</span><span class="p">;</span> <span class="p">}</span> <span class="nx">init</span><span class="p">().</span><span class="k">catch</span><span class="p">(</span><span class="nx">console</span><span class="p">.</span><span class="nx">err</span><span class="p">);</span> <span class="nx">module</span><span class="p">.</span><span class="nx">exports</span> <span class="o">=</span> <span class="p">{</span> <span class="na">get</span><span class="p">:</span> <span class="kd">function</span> <span class="p">(</span><span class="nx">key</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="nx">config</span><span class="p">[</span><span class="nx">key</span><span class="p">];</span> <span class="p">}</span> <span class="p">}</span> </code></pre></div></div> <p>其它模块在最顶层拿到的 process.globalConfig 的值会变成 undefined,上面第二种使用场景中的 exports.obj 将会构造失败,并且其他模块在最外层通过 config.get() 拿配置也会拿不到。</p> <h2 id="id-问题解决">问题解决</h2> <p>这个问题更通用的描述是:我们如何让一个模块在异步请求结束后,才能允许其他模块对他进行加载。最先想到了 top-level-await 特性,它可能是一个解决方案,但我没有深入调研,因为我们的工程不支持 mjs。<br /> 当应用加载时,首先加载的是 config 模块,如果 config 模块不加载完成,可能会影响到其他模块的加载,比如数据库连接池的初始化等等。在其他语言里,config 对象的初始化中的 IO 默认是同步的,但在 nodejs 中 IO 是异步的,会产生这个尴尬的问题。</p> <h3 id="id-双进程法">双进程法</h3> <p>一种解决方案是,在服务进程启动前,让另一个进程把配置解析好,比如写到文件里或者环境变量里。这样我们的 nodejs 进程就不需要在获取 config 时发生异步 IO 操作了,只需要同步读文件或环境变量。</p> <h3 id="id-双生命周期法">双生命周期法</h3> <p>跟<a href="https://github.com/zhangxiang958">翔哥</a>讨论了一波后,他说我应该换一个思路,Config 应该是一个独立的声明周期,当它变为加载完成的状态时,再去启动 App 的声明周期。</p> <p>我听到这个解法后,豁然开朗,这跟双进程法的思路是一样的,只不过这个做法是在单进程中控制生命周期的顺序。于是我对 config.js 进行了如下改造:</p> <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// config.js</span> <span class="kd">class</span> <span class="nx">Config</span> <span class="kd">extends</span> <span class="nx">Emitter</span> <span class="p">{</span> <span class="kd">constructor</span><span class="p">()</span> <span class="p">{</span> <span class="k">super</span><span class="p">();</span> <span class="k">this</span><span class="p">.</span><span class="nx">config</span> <span class="o">=</span> <span class="p">{};</span> <span class="nx">init</span><span class="p">().</span><span class="nx">then</span><span class="p">(</span><span class="nx">_</span> <span class="o">=&gt;</span> <span class="p">{</span> <span class="k">this</span><span class="p">.</span><span class="nx">emit</span><span class="p">(</span><span class="dl">'</span><span class="s1">ready</span><span class="dl">'</span><span class="p">);</span> <span class="p">}).</span><span class="k">catch</span><span class="p">(</span><span class="nx">err</span> <span class="o">=&gt;</span> <span class="p">{</span> <span class="nx">console</span><span class="p">.</span><span class="nx">err</span><span class="p">(</span><span class="nx">err</span><span class="p">);</span> <span class="nx">process</span><span class="p">.</span><span class="nx">exit</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">);</span> <span class="p">})</span> <span class="p">}</span> <span class="k">async</span> <span class="nx">init</span><span class="p">()</span> <span class="p">{</span> <span class="k">this</span><span class="p">.</span><span class="nx">config</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">initConfig</span><span class="p">();</span> <span class="p">}</span> <span class="kd">get</span><span class="p">(</span><span class="nx">key</span><span class="p">)</span> <span class="p">{</span> <span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">config</span><span class="p">[</span><span class="nx">key</span><span class="p">];</span> <span class="p">}</span> <span class="p">}</span> <span class="kd">let</span> <span class="nx">config</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">globalConfig</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Config</span><span class="p">();</span> <span class="nx">modules</span><span class="p">.</span><span class="k">export</span> <span class="o">=</span> <span class="nx">config</span><span class="p">;</span> </code></pre></div></div> <p>以 koa 应用举例,在 app.js 中,我们加载 config 后,监听他的 ready 事件,当 config 构建好之后,我们再去启动 App 的声明周期,此后所有由 App 带来的后续操作,一定可以访问到 config,且对已经写好的业务逻辑没有侵入性,因为 config 对象仍然有一个 get 方法。</p> <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">App</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">./app</span><span class="dl">'</span><span class="p">);</span> <span class="kd">const</span> <span class="nx">config</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">./config</span><span class="dl">'</span><span class="p">);</span> <span class="nx">config</span><span class="p">.</span><span class="nx">on</span><span class="p">(</span><span class="dl">'</span><span class="s1">ready</span><span class="dl">'</span><span class="p">,</span> <span class="kd">function</span><span class="p">()</span> <span class="p">{</span> <span class="kd">const</span> <span class="nx">app</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">App</span><span class="p">();</span> <span class="nx">app</span><span class="p">.</span><span class="nx">start</span><span class="p">();</span> <span class="p">});</span> </code></pre></div></div> 2021-05-20T00:00:00+00:00 https://ysmull.cn/blog/async-config.html https://ysmull.cn/blog/async-config.html 从一个线上问题谈谈查询条件的顺序的影响 <ul id="markdown-toc"> <li><a href="#id-问题背景" id="markdown-toc-id-问题背景">问题背景</a></li> <li><a href="#id-delete-和-select-的不同" id="markdown-toc-id-delete-和-select-的不同">delete 和 select 的不同</a></li> <li><a href="#id-where-条件顺序的影响" id="markdown-toc-id-where-条件顺序的影响">where 条件顺序的影响</a></li> <li><a href="#id-修复" id="markdown-toc-id-修复">修复</a></li> </ul> <h2 id="id-问题背景">问题背景</h2> <p>上班上得好好的,突然同事跟我说,在开发环境,有数无法创建自定义 SQL 了。根据日志定位到代码,发现相关代码很久没有改动过了,但是线上没有问题。报错的是这样一段 SQL:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">delete</span> <span class="k">from</span> <span class="n">global_parameter_ref</span> <span class="k">where</span> <span class="n">global_parameter_ref</span><span class="p">.</span><span class="n">resource_id</span> <span class="o">=</span> <span class="mi">5471</span> <span class="k">and</span> <span class="n">global_parameter_ref</span><span class="p">.</span><span class="n">resource_type</span> <span class="o">=</span> <span class="s1">'CUSTOM_TABLE'</span><span class="p">;</span> </code></pre></div></div> <p>执行上面这段 SQL,会报如下错误:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[22001][1292] Data truncation: Truncated incorrect DOUBLE value: 'c-1-132062-142229-kk6nz0ok' </code></pre></div></div> <p>因为 resource_id 的类型是 VARCHAR,报错信息说这里有一个 CAST 失败了,所以怀疑是不是使用了严格模式,遂写了如下查询 SQL 进行执行:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">select</span> <span class="o">*</span> <span class="k">from</span> <span class="n">global_parameter_ref</span> <span class="k">where</span> <span class="n">global_parameter_ref</span><span class="p">.</span><span class="n">resource_id</span> <span class="o">=</span> <span class="mi">5471</span> <span class="k">and</span> <span class="n">global_parameter_ref</span><span class="p">.</span><span class="n">resource_type</span> <span class="o">=</span> <span class="s1">'CUSTOM_TABLE'</span><span class="p">;</span> </code></pre></div></div> <p>把 delete 改成 select * 之后,执行起来没有任何问题。同事说,最近我们的 ORM 框架有一个<a href="https://github.com/ZhangDianPeng/dborm-mysql/pull/7/commits/d512513b5f197ff2811ad3e933073e0392fc64a4">改动</a>,会影响生成的 where 条件的顺序。 改动前生成的 SQL 如下,可以正常执行:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">select</span> <span class="o">*</span> <span class="k">from</span> <span class="n">global_parameter_ref</span> <span class="k">where</span> <span class="n">global_parameter_ref</span><span class="p">.</span><span class="n">resource_type</span> <span class="o">=</span> <span class="s1">'CUSTOM_TABLE'</span> <span class="k">and</span> <span class="n">global_parameter_ref</span><span class="p">.</span><span class="n">resource_id</span> <span class="o">=</span> <span class="mi">5471</span><span class="p">;</span> </code></pre></div></div> <p>那么问题就来了,为什么同样的一个操作,where 条件的顺序不一样,可以导致一条 SQL 报错,另一条 SQL 不报错呢?</p> <p>这里我们猜测,是因为筛选顺序不一致导致的。因为 resource_type = ‘CUSTOM_TABLE’ 的记录的 resource_id 的确是可以安全的 CAST 到数字的。如果 MySQL 先用 resource_type 进行筛选,就不会发生运行时类型转换错误。而如果先用 resource_id 进行筛选,某些记录的 resource_id 无法转换成数字,就会报错。</p> <p>这里有两个问题:</p> <ol> <li>为什么 SELECT 的时候不会报这个 CAST 错误?</li> <li>WHERE 条件的顺序一定是从左往右执行的吗?</li> </ol> <h2 id="id-delete-和-select-的不同">delete 和 select 的不同</h2> <p>通过试验发现, 对于一个 where 从句 where t.a = b,其中t.a 是表 t 的字段,b是字面量,如果字段类型和字面量类型不同,那么:</p> <ol> <li>在 delete 时,MySQL 会尝试把 a CAST 到 b 的类型</li> <li>在 select 时,MySQL 会尝试把 b CAST 到 a 的类型</li> </ol> <p>之所以这样做,一个比较合理的解释是:<strong>删除的时候,以字面量的类型为准,可以防止误删;查询时以字段类型为准,防止漏查</strong>。</p> <h2 id="id-where-条件顺序的影响">where 条件顺序的影响</h2> <p>我认为程序员在写 SQL 的时候,不应该去关心 where 条件的顺序,因为这个顺序不影响 SQL 的语义,具体数据库按怎样的顺序来执行这段 SQL,取决于数据库的实现或优化策略。</p> <p>如果两个筛选条件出现的顺序极大的影响了 SQL 的执行性能,那么你应该考虑的是,是不是应该给其中一个条件使用到的字段加索引,或者明确的强制这段 SQL 走这个索引。</p> <p>在我们上面的这个案例当中,where 从句的两个条件使用的字段都没有加索引,所以 MySQL 选择了 <strong>从左往右</strong> 的顺序来进行筛选,因此导致了 SQL 执行报错,这也是为什么当我们交换 where 条件的顺序后,就不报错了。</p> <p>当我们在 where 子句中增加一个主键筛选后,MySQL 就不会无脑的从左往右执行,而是走了主键索引,此时同样不会报错了:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">delete</span> <span class="k">from</span> <span class="n">global_parameter_ref</span> <span class="k">where</span> <span class="n">global_parameter_ref</span><span class="p">.</span><span class="n">resource_id</span> <span class="o">=</span> <span class="mi">5471</span> <span class="k">and</span> <span class="n">global_parameter_ref</span><span class="p">.</span><span class="n">resource_type</span> <span class="o">=</span> <span class="s1">'CUSTOM_TABLE'</span> <span class="k">and</span> <span class="n">id</span> <span class="o">=</span> <span class="mi">6</span><span class="p">;</span> <span class="c1">-- id 等于 6 的这条记录的 resource_id 是一个数字字符串,所以不会报错</span> </code></pre></div></div> <p>那么,如果我们给 resource_id 或者 custom_type 加一个索引,让 delete 语句强制走索引,这样不论 where 条件的顺序如何,这条 SQL 都不会报错。但可惜 MySQL 的 force index 语法并不支持 delete 语句。</p> <h2 id="id-修复">修复</h2> <p>在执行删除操作时,不要让数据库做隐式类型转换,在应用层生成 SQL 时处理。</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">delete</span> <span class="k">from</span> <span class="n">global_parameter_ref</span> <span class="k">where</span> <span class="n">global_parameter_ref</span><span class="p">.</span><span class="n">resource_id</span> <span class="o">=</span> <span class="s1">'5471'</span> <span class="c1">-- &lt;--明确类型</span> <span class="k">and</span> <span class="n">global_parameter_ref</span><span class="p">.</span><span class="n">resource_type</span> <span class="o">=</span> <span class="s1">'CUSTOM_TABLE'</span><span class="p">;</span> </code></pre></div></div> 2021-04-28T00:00:00+00:00 https://ysmull.cn/blog/where-clause-order.html https://ysmull.cn/blog/where-clause-order.html Java 中如何做类隔离 <ul id="markdown-toc"> <li><a href="#id-准备被依赖模块" id="markdown-toc-id-准备被依赖模块">准备被依赖模块</a> <ul> <li><a href="#id-模块a-10" id="markdown-toc-id-模块a-10">模块A 1.0</a></li> <li><a href="#id-模块a-20" id="markdown-toc-id-模块a-20">模块A 2.0</a></li> <li><a href="#id-模块b-10" id="markdown-toc-id-模块b-10">模块B 1.0</a></li> <li><a href="#id-模块b-20" id="markdown-toc-id-模块b-20">模块B 2.0</a></li> </ul> </li> <li><a href="#id-使用-maven-加载依赖" id="markdown-toc-id-使用-maven-加载依赖">使用 Maven 加载依赖</a></li> <li><a href="#id-使用类加载器加载多版本" id="markdown-toc-id-使用类加载器加载多版本">使用类加载器加载多版本</a></li> </ul> <h2 id="id-准备被依赖模块">准备被依赖模块</h2> <h3 id="id-模块a-10">模块A 1.0</h3> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">com.youdata</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">A</span> <span class="o">{</span> <span class="kd">static</span> <span class="o">{</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"A 1.0 loaded"</span><span class="o">);</span> <span class="o">}</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">printVersion</span><span class="o">()</span> <span class="o">{</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"I am A 1.0"</span><span class="o">);</span> <span class="o">}</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">printDepVersion</span><span class="o">()</span> <span class="o">{</span> <span class="no">B</span><span class="o">.</span><span class="na">printVersion</span><span class="o">();</span> <span class="o">}</span> <span class="o">}</span> </code></pre></div></div> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;dependencies&gt;</span> <span class="nt">&lt;dependency&gt;</span> <span class="nt">&lt;groupId&gt;</span>com.youdata<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>B<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>1.0<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;/dependency&gt;</span> <span class="nt">&lt;/dependencies&gt;</span> </code></pre></div></div> <h3 id="id-模块a-20">模块A 2.0</h3> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">com.youdata</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">A</span> <span class="o">{</span> <span class="kd">static</span> <span class="o">{</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"A 2.0 loaded"</span><span class="o">);</span> <span class="o">}</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">printVersion</span><span class="o">()</span> <span class="o">{</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"I am A 2.0"</span><span class="o">);</span> <span class="o">}</span> <span class="c1">// 新版通过反射调用</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">printDepVersion</span><span class="o">()</span> <span class="kd">throws</span> <span class="nc">NoSuchMethodException</span><span class="o">,</span> <span class="nc">InvocationTargetException</span><span class="o">,</span> <span class="nc">IllegalAccessException</span> <span class="o">{</span> <span class="nc">Method</span> <span class="n">printVersionNew</span> <span class="o">=</span> <span class="no">B</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getMethod</span><span class="o">(</span><span class="s">"printVersionNew"</span><span class="o">);</span> <span class="n">printVersionNew</span><span class="o">.</span><span class="na">invoke</span><span class="o">(</span><span class="kc">null</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> </code></pre></div></div> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;dependencies&gt;</span> <span class="nt">&lt;dependency&gt;</span> <span class="nt">&lt;groupId&gt;</span>com.youdata<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>B<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>1.0<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;/dependency&gt;</span> <span class="nt">&lt;/dependencies&gt;</span> </code></pre></div></div> <h3 id="id-模块b-10">模块B 1.0</h3> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">com.youdata</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">B</span> <span class="o">{</span> <span class="kd">static</span> <span class="o">{</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"B 1.0 loaded"</span><span class="o">);</span> <span class="o">}</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">printVersion</span><span class="o">()</span> <span class="o">{</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"I am B 1.0"</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> </code></pre></div></div> <h3 id="id-模块b-20">模块B 2.0</h3> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">com.youdata</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">B</span> <span class="o">{</span> <span class="kd">static</span> <span class="o">{</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"B 2.0 loaded"</span><span class="o">);</span> <span class="o">}</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">printVersionNew</span><span class="o">()</span> <span class="o">{</span> <span class="c1">// 方法名改了</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"I am B 2.0"</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> </code></pre></div></div> <h2 id="id-使用-maven-加载依赖">使用 Maven 加载依赖</h2> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">com.youdata</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">Main</span> <span class="o">{</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> <span class="no">A</span><span class="o">.</span><span class="na">printVersion</span><span class="o">();</span> <span class="no">A</span><span class="o">.</span><span class="na">printDepVersion</span><span class="o">();</span> <span class="o">}</span> <span class="o">}</span> </code></pre></div></div> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;dependencies&gt;</span> <span class="nt">&lt;dependency&gt;</span> <span class="nt">&lt;groupId&gt;</span>com.youdata<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>A<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>1.0<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;/dependency&gt;</span> <span class="nt">&lt;/dependencies&gt;</span> </code></pre></div></div> <p>我们运行一下:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A 1.0 loaded I am A 1.0 Exception in thread "main" java.lang.NoClassDefFoundError: com/youdata/B at com.youdata.A.printDepVersion(A.java:14) at com.youdata.Main.main(Main.java:7) Caused by: java.lang.ClassNotFoundException: com.youdata.B at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 2 more </code></pre></div></div> <p>报错原因是因为 A 在执行 B 的静态方法时,找不到类 B,从而无法加载类 B,我们在 pom.xml 中引入 B 这个依赖,并重新运行:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;dependencies&gt;</span> <span class="nt">&lt;dependency&gt;</span> <span class="nt">&lt;groupId&gt;</span>com.youdata<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>A<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>1.0<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;/dependency&gt;</span> <span class="nt">&lt;dependency&gt;</span> <span class="nt">&lt;groupId&gt;</span>com.youdata<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>B<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>1.0<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;/dependency&gt;</span> <span class="nt">&lt;/dependencies&gt;</span> </code></pre></div></div> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A 1.0 loaded I am A 1.0 B 1.0 loaded I am B 1.0 </code></pre></div></div> <p>如果我们把 B 的依赖版本升级到 2.0,重新运行时,A 会调用 B 2.0 版本的静态方法:</p> <div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">&lt;dependencies&gt;</span> <span class="nt">&lt;dependency&gt;</span> <span class="nt">&lt;groupId&gt;</span>com.youdata<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>A<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>1.0<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;/dependency&gt;</span> <span class="nt">&lt;dependency&gt;</span> <span class="nt">&lt;groupId&gt;</span>com.youdata<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>B<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>2.0<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;/dependency&gt;</span> <span class="nt">&lt;/dependencies&gt;</span> </code></pre></div></div> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A 1.0 loaded I am A 1.0 B 2.0 loaded I am B 2.0 </code></pre></div></div> <p>可以看到,使用 Maven 的方式,调用方声明自己需要一个 A 模块,如果不提供 B 模块的任何一个版本,A 模块是无法正常工作的。</p> <p>当我们提供 1.0 版本的 B 模块,A 就是用 1.0 的 B 模块工作,当我们提供 2.0 版本的 B 模块,A 就是用 2.0 的 B 模块工作。</p> <h2 id="id-使用类加载器加载多版本">使用类加载器加载多版本</h2> <p>我们改造调用方的项目结构如下,使用 URLClassLoader 同时加载指定路径的所有 jar 包。</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>. ├── pom.xml └── src └── main ├── java │   └── com │   └── youdata │   └── Main.java └── resources └── lib ├── 1.0 │   ├── A-1.0.jar │   └── B-1.0.jar └── 2.0 ├── A-2.0.jar └── B-2.0.jar </code></pre></div></div> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">package</span> <span class="nn">com.youdata</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.lang.reflect.Method</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.net.MalformedURLException</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.net.URL</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.net.URLClassLoader</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.nio.file.Files</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.nio.file.Path</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.nio.file.Paths</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.Objects</span><span class="o">;</span> <span class="kn">import</span> <span class="nn">java.util.stream.Stream</span><span class="o">;</span> <span class="kd">public</span> <span class="kd">class</span> <span class="nc">Main</span> <span class="o">{</span> <span class="kd">public</span> <span class="kd">static</span> <span class="nc">Class</span><span class="o">&lt;?&gt;</span> <span class="n">loadClassA</span><span class="o">(</span><span class="nc">String</span> <span class="n">jarPath</span><span class="o">,</span> <span class="nc">ClassLoader</span> <span class="n">loader</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">Exception</span> <span class="o">{</span> <span class="nc">String</span> <span class="n">lib</span> <span class="o">=</span> <span class="nc">Main</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getClassLoader</span><span class="o">().</span><span class="na">getResource</span><span class="o">(</span><span class="n">jarPath</span><span class="o">).</span><span class="na">getPath</span><span class="o">();</span> <span class="nc">Stream</span><span class="o">&lt;</span><span class="nc">Path</span><span class="o">&gt;</span> <span class="n">walk</span> <span class="o">=</span> <span class="nc">Files</span><span class="o">.</span><span class="na">walk</span><span class="o">(</span><span class="nc">Paths</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">lib</span><span class="o">));</span> <span class="no">URL</span><span class="o">[]</span> <span class="n">jars</span> <span class="o">=</span> <span class="n">walk</span><span class="o">.</span><span class="na">filter</span><span class="o">(</span><span class="n">f</span> <span class="o">-&gt;</span> <span class="n">f</span><span class="o">.</span><span class="na">getFileName</span><span class="o">().</span><span class="na">toString</span><span class="o">().</span><span class="na">endsWith</span><span class="o">(</span><span class="s">".jar"</span><span class="o">))</span> <span class="o">.</span><span class="na">map</span><span class="o">(</span><span class="n">p</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="k">try</span> <span class="o">{</span> <span class="k">return</span> <span class="n">p</span><span class="o">.</span><span class="na">toUri</span><span class="o">().</span><span class="na">toURL</span><span class="o">();</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">MalformedURLException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span> <span class="k">return</span> <span class="kc">null</span><span class="o">;</span> <span class="o">}</span> <span class="o">}).</span><span class="na">filter</span><span class="o">(</span><span class="nl">Objects:</span><span class="o">:</span><span class="n">nonNull</span><span class="o">).</span><span class="na">toArray</span><span class="o">(</span><span class="no">URL</span><span class="o">[]::</span><span class="k">new</span><span class="o">);</span> <span class="nc">URLClassLoader</span> <span class="n">classLoader</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">URLClassLoader</span><span class="o">(</span><span class="n">jars</span><span class="o">,</span> <span class="n">loader</span><span class="o">);</span> <span class="k">return</span> <span class="n">classLoader</span><span class="o">.</span><span class="na">loadClass</span><span class="o">(</span><span class="s">"com.youdata.A"</span><span class="o">);</span> <span class="o">}</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">invokeAsMethod</span><span class="o">(</span><span class="nc">Class</span><span class="o">&lt;?&gt;</span> <span class="n">aClass</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">Exception</span> <span class="o">{</span> <span class="nc">Method</span> <span class="n">printVersion</span> <span class="o">=</span> <span class="n">aClass</span><span class="o">.</span><span class="na">getMethod</span><span class="o">(</span><span class="s">"printVersion"</span><span class="o">);</span> <span class="n">printVersion</span><span class="o">.</span><span class="na">invoke</span><span class="o">(</span><span class="kc">null</span><span class="o">);</span> <span class="nc">Method</span> <span class="n">printDepVersion</span> <span class="o">=</span> <span class="n">aClass</span><span class="o">.</span><span class="na">getMethod</span><span class="o">(</span><span class="s">"printDepVersion"</span><span class="o">);</span> <span class="n">printDepVersion</span><span class="o">.</span><span class="na">invoke</span><span class="o">(</span><span class="kc">null</span><span class="o">);</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">();</span> <span class="o">}</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">loadClassInNewThread</span><span class="o">(</span><span class="nc">String</span> <span class="n">jarPath</span><span class="o">)</span> <span class="o">{</span> <span class="k">new</span> <span class="nf">Thread</span><span class="o">(()</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="k">try</span> <span class="o">{</span> <span class="nc">Class</span><span class="o">&lt;?&gt;</span> <span class="n">aClass</span> <span class="o">=</span> <span class="n">loadClassA</span><span class="o">(</span><span class="n">jarPath</span><span class="o">,</span> <span class="nc">Thread</span><span class="o">.</span><span class="na">currentThread</span><span class="o">().</span><span class="na">getContextClassLoader</span><span class="o">());</span> <span class="k">while</span> <span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span> <span class="n">invokeAsMethod</span><span class="o">(</span><span class="n">aClass</span><span class="o">);</span> <span class="nc">Thread</span><span class="o">.</span><span class="na">sleep</span><span class="o">(</span><span class="mi">2000</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span> <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span> <span class="o">}</span> <span class="o">}).</span><span class="na">start</span><span class="o">();</span> <span class="o">}</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="kd">throws</span> <span class="nc">InterruptedException</span> <span class="o">{</span> <span class="n">loadClassInNewThread</span><span class="o">(</span><span class="s">"lib/1.0"</span><span class="o">);</span> <span class="nc">Thread</span><span class="o">.</span><span class="na">sleep</span><span class="o">(</span><span class="mi">1000</span><span class="o">);</span> <span class="c1">// 时间上错位打印</span> <span class="n">loadClassInNewThread</span><span class="o">(</span><span class="s">"lib/2.0"</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> </code></pre></div></div> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>A 1.0 loaded I am A 1.0 B 1.0 loaded I am B 1.0 A 2.0 loaded I am A 2.0 B 2.0 loaded I am B 2.0 I am A 1.0 I am B 1.0 I am A 2.0 I am B 2.0 I am A 1.0 I am B 1.0 I am A 2.0 I am B 2.0 I am A 1.0 I am B 1.0 ... </code></pre></div></div> <p>我们可以把 jar 包修改一下,让 2.0 目录下的 A-2.0.jar 使用 B-1.0.jar:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>. ├── pom.xml └── src └── main ├── java │   └── com │   └── youdata │   └── Main.java └── resources └── lib ├── 1.0 │   ├── A-1.0.jar │   └── B-1.0.jar └── 2.0 ├── A-2.0.jar └── B-1.0.jar &lt;====== 修改为 B-1.0.jar </code></pre></div></div> <p>重新执行代码报了 <code class="language-plaintext highlighter-rouge">NoSuchMethodException</code> 错误:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.youdata.Main.invokeAsMethod(Main.java:34) at com.youdata.Main.lambda$loadClassInNewThread$3(Main.java:43) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoSuchMethodException: com.youdata.B.printVersionNew() at java.lang.Class.getMethod(Class.java:1786) at com.youdata.A.printDepVersion(A.java:17) ... 7 more </code></pre></div></div> <p>并且实验还发现:</p> <ol> <li>把 <code class="language-plaintext highlighter-rouge">Thread.currentThread().getContextClassLoader()</code> 换成 <code class="language-plaintext highlighter-rouge">Main.class.getClassLoader()</code> 甚至换成 <code class="language-plaintext highlighter-rouge">null</code> 也是 work 的。</li> <li>去掉 sleep,每一次循环都重新加载类并且,两个线程并发运行,也不会发生调用错误,这说明被加载的类没有相互覆盖。</li> </ol> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">loadClassInNewThread</span><span class="o">(</span><span class="nc">String</span> <span class="n">jarPath</span><span class="o">)</span> <span class="o">{</span> <span class="k">new</span> <span class="nf">Thread</span><span class="o">(()</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="k">try</span> <span class="o">{</span> <span class="k">while</span> <span class="o">(</span><span class="kc">true</span><span class="o">)</span> <span class="o">{</span> <span class="nc">Class</span><span class="o">&lt;?&gt;</span> <span class="n">aClass</span> <span class="o">=</span> <span class="n">loadClassA</span><span class="o">(</span><span class="n">jarPath</span><span class="o">,</span> <span class="kc">null</span><span class="o">);</span> <span class="n">invokeAsMethod</span><span class="o">(</span><span class="n">aClass</span><span class="o">);</span> <span class="o">}</span> <span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="nc">Exception</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span> <span class="n">e</span><span class="o">.</span><span class="na">printStackTrace</span><span class="o">();</span> <span class="o">}</span> <span class="o">}).</span><span class="na">start</span><span class="o">();</span> <span class="o">}</span> </code></pre></div></div> 2021-04-13T19:35:00+00:00 https://ysmull.cn/blog/classLoader.html https://ysmull.cn/blog/classLoader.html Thread Local 之我见 <ul id="markdown-toc"> <li><a href="#id-引子" id="markdown-toc-id-引子">引子</a></li> <li><a href="#id-threadlocalmap-的实现" id="markdown-toc-id-threadlocalmap-的实现">ThreadLocalMap 的实现</a></li> <li><a href="#id-注意事项" id="markdown-toc-id-注意事项">注意事项</a></li> </ul> <h2 id="id-引子">引子</h2> <p>我们有了一个线程</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Thread</span> <span class="n">t</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Thread</span><span class="o">();</span> </code></pre></div></div> <p>我们希望放置一些数据,使得在该线程的任何地方都可以访问和修改这些数据,这就是 ThreadLocal 对象。理所应当的,我们应该把这个些数据存放在当前 Thread 的对象上,这也就是为什么 Thread 类上有一个 threadLocals 字段,它的类型是 ThreadLocalMap,ThreadLocalMap 顾名思义是一个 Map,key 为 ThreadLocal 对象,value 是我们要存储的对象。</p> <div class="language-java line-numbers highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/* ThreadLocal values pertaining to this thread. This map is maintained * by the ThreadLocal class. */</span> <span class="nc">ThreadLocal</span><span class="o">.</span><span class="na">ThreadLocalMap</span> <span class="n">threadLocals</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span> </code></pre></div></div> <p>每一个 threadLocal 对象只能存放「一个东西」。如果你想存放苹果,再存放书,得分别为要存放的东西创建 ThreadLocal 对象:</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">ThreadLocal</span><span class="o">&lt;</span><span class="nc">Apple</span><span class="o">&gt;</span> <span class="n">appleThreadLocal</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ThreadLocal</span><span class="o">&lt;&gt;();</span> <span class="nc">ThreadLocal</span><span class="o">&lt;</span><span class="nc">Book</span><span class="o">&gt;</span> <span class="n">bookThreadLocal</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ThreadLocal</span><span class="o">&lt;&gt;();</span> </code></pre></div></div> <p>不过,到目前为止,仍然是什么特殊的事情都没有发生,因为 <strong>ThreadLocal 的构造函数是空的</strong> :</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/** * Creates a thread local variable. * @see #withInitial(java.util.function.Supplier) */</span> <span class="kd">public</span> <span class="nf">ThreadLocal</span><span class="o">()</span> <span class="o">{</span> <span class="o">}</span> </code></pre></div></div> <p>多个线程可以使用同一个 ThreadLocal 对象:</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="nc">ThreadLocal</span><span class="o">&lt;</span><span class="nc">Apple</span><span class="o">&gt;</span> <span class="n">appThreadLocal</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ThreadLocal</span><span class="o">&lt;&gt;();</span> <span class="c1">// thread1 // thread2</span> <span class="o">...</span> <span class="o">...</span> <span class="nc">Apple</span> <span class="n">apple1</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Apple</span><span class="o">();</span> <span class="nc">Apple</span> <span class="n">apple2</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Apple</span><span class="o">();</span> <span class="n">appThreadLocal</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">apple1</span><span class="o">);</span> <span class="n">appThreadLocal</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">apple2</span><span class="o">);</span> <span class="o">...</span> <span class="o">...</span> <span class="n">appThreadLocal</span><span class="o">.</span><span class="na">get</span><span class="o">();</span> <span class="c1">// apple1 appThreadLocal.get(); // apple2</span> </code></pre></div></div> <p>在整个 Thread.java 的源码中,只有 exit() 方法使用到了 threadLocals</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">private</span> <span class="kt">void</span> <span class="nf">exit</span><span class="o">()</span> <span class="o">{</span> <span class="o">...</span> <span class="cm">/* Speed the release of some of these resources */</span> <span class="n">threadLocals</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span> <span class="o">...</span> <span class="o">}</span> </code></pre></div></div> <p>而 threadLocalMap 的初始化,是在第一个 ThreadLocal 对象调用 set 或 get 的时候发生的</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/** * Create the map associated with a ThreadLocal. Overridden in * InheritableThreadLocal. * * @param t the current thread * @param firstValue value for the initial entry of the map */</span> <span class="kt">void</span> <span class="nf">createMap</span><span class="o">(</span><span class="nc">Thread</span> <span class="n">t</span><span class="o">,</span> <span class="no">T</span> <span class="n">firstValue</span><span class="o">)</span> <span class="o">{</span> <span class="n">t</span><span class="o">.</span><span class="na">threadLocals</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ThreadLocalMap</span><span class="o">(</span><span class="k">this</span><span class="o">,</span> <span class="n">firstValue</span><span class="o">);</span> <span class="o">}</span> </code></pre></div></div> <p>我们来看一下set方法是如何执行的</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/** * Sets the current thread's copy of this thread-local variable * to the specified value. Most subclasses will have no need to * override this method, relying solely on the {@link #initialValue} * method to set the values of thread-locals. * * @param value the value to be stored in the current thread's copy of * this thread-local. */</span> <span class="kd">public</span> <span class="kt">void</span> <span class="nf">set</span><span class="o">(</span><span class="no">T</span> <span class="n">value</span><span class="o">)</span> <span class="o">{</span> <span class="nc">Thread</span> <span class="n">t</span> <span class="o">=</span> <span class="nc">Thread</span><span class="o">.</span><span class="na">currentThread</span><span class="o">();</span> <span class="nc">ThreadLocalMap</span> <span class="n">map</span> <span class="o">=</span> <span class="n">getMap</span><span class="o">(</span><span class="n">t</span><span class="o">);</span> <span class="k">if</span> <span class="o">(</span><span class="n">map</span> <span class="o">!=</span> <span class="kc">null</span><span class="o">)</span> <span class="n">map</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="k">this</span><span class="o">,</span> <span class="n">value</span><span class="o">);</span> <span class="k">else</span> <span class="nf">createMap</span><span class="o">(</span><span class="n">t</span><span class="o">,</span> <span class="n">value</span><span class="o">);</span> <span class="o">}</span> </code></pre></div></div> <p>从 <code class="language-plaintext highlighter-rouge">map.set(this, value);</code> 可以看到,我们把要存储的值放到了 key 为当前 ThreadLocal 对象的 ThreadLocalMap 中去了。取值的时候也是从这个 map 中取,所以问题的关键就在于 ThreadLocalMap 的实现了。</p> <h2 id="id-threadlocalmap-的实现">ThreadLocalMap 的实现</h2> <p>跟 HashMap 类似,也是用数组来实现的</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">private</span> <span class="nc">Entry</span><span class="o">[]</span> <span class="n">table</span><span class="o">;</span> </code></pre></div></div> <p>然而这里的Entry 是 WeakReference 的子类</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cm">/** * The entries in this hash map extend WeakReference, using * its main ref field as the key (which is always a * ThreadLocal object). Note that null keys (i.e. entry.get() * == null) mean that the key is no longer referenced, so the * entry can be expunged from table. Such entries are referred to * as "stale entries" in the code that follows. */</span> <span class="kd">static</span> <span class="kd">class</span> <span class="nc">Entry</span> <span class="kd">extends</span> <span class="nc">WeakReference</span><span class="o">&lt;</span><span class="nc">ThreadLocal</span><span class="o">&lt;?&gt;&gt;</span> <span class="o">{</span> <span class="cm">/** The value associated with this ThreadLocal. */</span> <span class="nc">Object</span> <span class="n">value</span><span class="o">;</span> <span class="nc">Entry</span><span class="o">(</span><span class="nc">ThreadLocal</span><span class="o">&lt;?&gt;</span> <span class="n">k</span><span class="o">,</span> <span class="nc">Object</span> <span class="n">v</span><span class="o">)</span> <span class="o">{</span> <span class="kd">super</span><span class="o">(</span><span class="n">k</span><span class="o">);</span> <span class="n">value</span> <span class="o">=</span> <span class="n">v</span><span class="o">;</span> <span class="o">}</span> <span class="o">}</span> </code></pre></div></div> <p>由于弱引用的特性,当 ThreadLocal 对象不可达的时候,ThreadLocalMap 的 key 就会被 GC 回收掉。试想,如果这里不是弱引用,而是强引用,那么 ThreadLocal 对象将永远不会被回收,除非线程终止。因为 Thread 持有 threadLocals,threadLocals 的 Entry 因为不是弱引用,就会持有 threadLocal 对象的强引用,如果不显式调用 ThreadLocal 的 remove 去掉这里的强引用,<strong>线程生命周期内,threadLocal 就不会被回收</strong>。当线程结束后,Thread 的 exit() 方法会执行 <code class="language-plaintext highlighter-rouge">threadLocals = null</code> 使得线程内所有的 ThreadLocal 对象以及其上面带有的 value 不可达,导致被回收。不过 exit 方法的注释上说,这里只是为了<em>加速</em> 资源的释放,即便线程执行 exit 的时候不释放,线程本身最终也会不可达,所有线程相关的资源最终还是会被回收。</p> <p><strong>我再画一幅图解释一下:(人类的本质是复读机!)</strong></p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">Thread</span> <span class="n">t</span> <span class="c1">// 某一个线程</span> <span class="err">↓</span> <span class="nc">ThreadLocal</span><span class="o">.</span><span class="na">ThreadLocalMap</span> <span class="n">t</span><span class="o">.</span><span class="na">threadLocals</span> <span class="c1">// 线程持有的 Map 对象</span> <span class="err">↓</span> <span class="nc">Entry</span> <span class="n">table</span><span class="o">[</span><span class="n">i</span><span class="o">]</span> <span class="c1">// Map 持有的某一个 Entry 对象</span> </code></pre></div></div> <ol> <li> <p>假设 Entry 持有 key 和 value 的强引用,那么如果 ThreadLocalMap 不自己清理,key 和 value 就会伴随整个 Thread 的生命周期。</p> </li> <li> <p>假设 Entry 是 key 的弱引用,那么 key 不可达时(也就是 threadLocal 不可达时),key 会被 gc,<strong>value 仍然伴随 Thread 的生命周期。</strong></p> </li> </ol> <p>那么 ThreadLocal 对象在什么时候才会不可达呢?最简单的情况是,离开了函数调用栈空间了,我们来看下面这个例子</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">test</span><span class="o">()</span> <span class="o">{</span> <span class="nc">ThreadLocal</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">&gt;</span> <span class="n">t</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ThreadLocal</span><span class="o">&lt;&gt;();</span> <span class="n">t</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="s">"abc"</span><span class="o">);</span> <span class="o">}</span> <span class="kd">public</span> <span class="kd">static</span> <span class="kt">void</span> <span class="nf">main</span><span class="o">(</span><span class="nc">String</span><span class="o">[]</span> <span class="n">args</span><span class="o">)</span> <span class="o">{</span> <span class="k">new</span> <span class="nf">Thread</span><span class="o">(()</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="n">test</span><span class="o">();</span> <span class="c1">// 1</span> <span class="nc">System</span><span class="o">.</span><span class="na">gc</span><span class="o">();</span> <span class="c1">// 2</span> <span class="o">}).</span><span class="na">start</span><span class="o">();</span> <span class="o">}</span> </code></pre></div></div> <p>在线程执行完 test() 方法后,执行 System.gc() 之前,也就是上述代码注释 1 的位置,当前线程的 threadLocals 对象如下:<br /> <img src="/img/2019-08-03-123120.png" alt="" /><br /> 可以看见 ThreadLocalMap 的 Entry 的 referent 和 value 都还在,当执行 System.gc() 后,我们看一下有什么变化:<br /> <img src="/img/2019-08-03-123316.png" alt="" /><br /> 此时由于 threadLocal 对象已经不可达,所以会被 gc 回收。然而,<strong>value 对象还没回收呢</strong>!</p> <p>怎样才能让 value 对象也可以回收呢?<strong>上面已经说过很多次了</strong>,value 对象被回收当且仅当:</p> <ol> <li>ThreadLocaMap 自己清理</li> <li>Thread 对象被回收</li> </ol> <p>下面我们来讲解 ThreadLocalMap 什么情况下才会自己清理 key 为 null 的 Entry</p> <p>在 ThreadLocalMap 类中 ,真正可以回收 value (以及 key) 的方法是 expungeStaleEntry(int staleSlot) 方法:</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">private</span> <span class="kt">int</span> <span class="nf">expungeStaleEntry</span><span class="o">(</span><span class="kt">int</span> <span class="n">staleSlot</span><span class="o">)</span> <span class="o">{</span> <span class="nc">Entry</span><span class="o">[]</span> <span class="n">tab</span> <span class="o">=</span> <span class="n">table</span><span class="o">;</span> <span class="kt">int</span> <span class="n">len</span> <span class="o">=</span> <span class="n">tab</span><span class="o">.</span><span class="na">length</span><span class="o">;</span> <span class="c1">// expunge entry at staleSlot</span> <span class="n">tab</span><span class="o">[</span><span class="n">staleSlot</span><span class="o">].</span><span class="na">value</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span> <span class="n">tab</span><span class="o">[</span><span class="n">staleSlot</span><span class="o">]</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span> <span class="n">size</span><span class="o">--;</span> <span class="c1">// Rehash until we encounter null</span> <span class="o">...</span> <span class="o">}</span> </code></pre></div></div> <p>这个方法的实现是比较复杂的,不仅直接删除指定 index 的元素,并且在 rehash 的过程中也会清理一些其它的元素。</p> <p>我们通过 IDE 分析调用链可以知道,只有在 ThreadLocal 的 get 、set、remove 方法被调用时,才有可能会进行清理。</p> <h2 id="id-注意事项">注意事项</h2> <p>我们使用 ThreadLocal 最常见的场景是把 ThreadLocal 作为一个类的静态字段(即便不是静态字段,在 IOC 环境下,也可能是个全局的单例对象)多个线程共用同一个 ThreadLocal 对象。(也就是使用同一个 ThreadLocal 对象作为 key 放到各自线程 ThreadLocalMap 里)。</p> <p>在这种场景下,ThreadLocal 对象的生命周期与承载它的对象的生命周期相同,<strong>在大多数 Web 场景下,这个 ThreadLocal 对象几乎在整个应用的生命周期中都是可达的</strong>。所以每个线程各自的 ThreadLocalMap 的 key 并不会因为使用了弱引用而被 GC 回收掉。我们只能够在每个线程中<strong>手动调用 ThreadLocal 的 remove 方法</strong>对 key 和 value 进行置 null,使得 GC 可以清理掉线程持有的 value(GC 清理不了 key,因为 key 总是可达)。如果不手动调用 remove,那么只有在线程被清理的时候,value 才会被清理(这里说了很多遍了,看不懂请重读上文)。</p> <p>同时,在 web 场景下,大多数 http server 会使用到线程池,如果在上一次请求中没有主动调用 remove 进行清理,线程池为下一次请求分配的可能是同一个线程,那么调用 ThreadLocal 的 get 方法拿到的就是上一次线程存储的值,这可能会导致错误的业务逻辑。</p> <p>因此,我们在使用 ThreadLocal 时,应该总是在当前线程结束前手动调用 remove 方法来清理这个线程的 ThreadLocalMap,否则,是有可能内存泄漏的(线程池中的每个线程的 threadLocals 都持有一个相同的 key 的 Entry,以及一个无用的 value)。</p> 2019-08-03T19:35:00+00:00 https://ysmull.cn/blog/thraedLocal.html https://ysmull.cn/blog/thraedLocal.html 「细谈」Java 中的弱引用(WeakReference) <ul id="markdown-toc"> <li><a href="#id-弱引用weak-reference" id="markdown-toc-id-弱引用weak-reference">弱引用(Weak Reference)</a> <ul> <li><a href="#id-几种典型场景" id="markdown-toc-id-几种典型场景">几种典型场景</a></li> </ul> </li> <li><a href="#id-referencequeue" id="markdown-toc-id-referencequeue">ReferenceQueue</a> <ul> <li><a href="#id-特殊情况" id="markdown-toc-id-特殊情况">特殊情况</a></li> </ul> </li> </ul> <p>本文只讲弱引用,文章中的内容均来自本人的实践的总结,某些观点不常见于其它文章。<br /> 该文是理解 ThreadLocal 的基础。</p> <h2 id="id-弱引用weak-reference">弱引用(Weak Reference)</h2> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">WeakReference</span><span class="o">&lt;</span><span class="no">T</span><span class="o">&gt;</span> <span class="n">weakRef</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">WeakReference</span><span class="o">&lt;&gt;(</span><span class="n">referent</span><span class="o">);</span> </code></pre></div></div> <p>当发生 gc 时, 如果 referent 对象满足下述条件则一定会被回收:</p> <ol> <li><code class="language-plaintext highlighter-rouge">referent</code> 没有强引用</li> <li><code class="language-plaintext highlighter-rouge">referent</code> 没有软引用</li> </ol> <h3 id="id-几种典型场景">几种典型场景</h3> <ul> <li>最简单的例子 <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">WeakReference</span><span class="o">&lt;</span><span class="nc">List</span><span class="o">&gt;</span> <span class="n">arrRef</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">WeakReference</span><span class="o">&lt;&gt;(</span><span class="nc">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span><span class="mi">2</span><span class="o">,</span><span class="mi">3</span><span class="o">));</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">arrRef</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="c1">// [1, 2, 3]</span> <span class="nc">System</span><span class="o">.</span><span class="na">gc</span><span class="o">();</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">arrRef</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="c1">// null</span> </code></pre></div> </div> </li> <li>有强引用无法回收 <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">List</span><span class="o">&lt;</span><span class="nc">Integer</span><span class="o">&gt;</span> <span class="n">arr</span> <span class="o">=</span> <span class="nc">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="mi">1</span><span class="o">,</span><span class="mi">2</span><span class="o">,</span><span class="mi">3</span><span class="o">);</span> <span class="nc">WeakReference</span><span class="o">&lt;</span><span class="nc">List</span><span class="o">&gt;</span> <span class="n">arrRef</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">WeakReference</span><span class="o">&lt;&gt;(</span><span class="n">arr</span><span class="o">);</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">arrRef</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="c1">// [1, 2, 3]</span> <span class="nc">System</span><span class="o">.</span><span class="na">gc</span><span class="o">();</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">arrRef</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="c1">// [1, 2, 3] 还有 arr 强引用,无法回收</span> <span class="n">arr</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span> <span class="nc">System</span><span class="o">.</span><span class="na">gc</span><span class="o">();</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">arrRef</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="c1">// null</span> </code></pre></div> </div> </li> <li>final 的东西无法回收 <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">WeakReference</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">&gt;</span> <span class="n">strRef1</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">WeakReference</span><span class="o">&lt;&gt;(</span><span class="k">new</span> <span class="nc">String</span><span class="o">(</span><span class="s">"abc"</span><span class="o">));</span> <span class="nc">System</span><span class="o">.</span><span class="na">gc</span><span class="o">();</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">strRef1</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="c1">// null</span> <span class="nc">WeakReference</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">&gt;</span> <span class="n">strRef2</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">WeakReference</span><span class="o">&lt;&gt;(</span><span class="s">"abc"</span><span class="o">);</span> <span class="nc">System</span><span class="o">.</span><span class="na">gc</span><span class="o">();</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">strRef2</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="c1">// abc</span> </code></pre></div> </div> </li> <li>WeakReference 数组内的元素会被回收(弱引用数组的每个 item 都是弱引用) <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">WeakReference</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">&gt;</span> <span class="n">a</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">WeakReference</span><span class="o">&lt;&gt;(</span><span class="k">new</span> <span class="nc">String</span><span class="o">(</span><span class="s">"aaa"</span><span class="o">));</span> <span class="nc">WeakReference</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">&gt;</span> <span class="n">b</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">WeakReference</span><span class="o">&lt;&gt;(</span><span class="k">new</span> <span class="nc">String</span><span class="o">(</span><span class="s">"bbb"</span><span class="o">));</span> <span class="nc">WeakReference</span><span class="o">[]</span> <span class="n">tab</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">WeakReference</span><span class="o">[]</span> <span class="o">{</span><span class="n">a</span><span class="o">,</span> <span class="n">b</span><span class="o">};</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">a</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="c1">// aaa</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">b</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="c1">// bbb</span> <span class="nc">System</span><span class="o">.</span><span class="na">gc</span><span class="o">();</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">a</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="c1">// null</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">b</span><span class="o">.</span><span class="na">get</span><span class="o">());</span> <span class="c1">// null</span> </code></pre></div> </div> </li> </ul> <h2 id="id-referencequeue">ReferenceQueue</h2> <p>用来监视被引用的对象是否已经被回收了。下面我们用 referenceQueue 来探究一下 WeakReference 的回收。</p> <p><strong><em>(为求代码清晰,下面的代码一律不捕获 <code class="language-plaintext highlighter-rouge">InterruptedException</code>)</em></strong></p> <p>首先创建一个 referenceQueue, 在另一个线程中调用 remove,该调用是阻塞调用,如果有引用被回收,那么会调用成功,返回该该引用的 this。<br /> 最简单的例子:</p> <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">ReferenceQueue</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">&gt;</span> <span class="n">referenceQueue</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ReferenceQueue</span><span class="o">&lt;&gt;();</span> <span class="c1">// 监视线程</span> <span class="k">new</span> <span class="nf">Thread</span><span class="o">(()</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="nc">Reference</span> <span class="n">remove</span> <span class="o">=</span> <span class="n">referenceQueue</span><span class="o">.</span><span class="na">remove</span><span class="o">();</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"remove:"</span> <span class="o">+</span> <span class="n">remove</span><span class="o">);</span> <span class="o">}).</span><span class="na">start</span><span class="o">();</span> <span class="nc">WeakReference</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">&gt;</span> <span class="n">ref</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">WeakReference</span><span class="o">&lt;&gt;(</span><span class="k">new</span> <span class="nc">String</span><span class="o">(</span><span class="s">"111"</span><span class="o">),</span> <span class="n">referenceQueue</span><span class="o">);</span> <span class="nc">System</span><span class="o">.</span><span class="na">gc</span><span class="o">();</span> <span class="c1">// remove:java.lang.ref.WeakReference@322b4b46</span> </code></pre></div></div> <h3 id="id-特殊情况">特殊情况</h3> <ol> <li>下例说明,<strong>不把 new 出来的 WeakReference 赋值给任何变量,那么可能虚拟机可能当场就回收了</strong>,因为 referenceQueue 并没有捕获到回收消息。 <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="mi">1</span> <span class="nc">ReferenceQueue</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">&gt;</span> <span class="n">referenceQueue</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ReferenceQueue</span><span class="o">&lt;&gt;();</span> <span class="c1">// 监视线程</span> <span class="k">new</span> <span class="nf">Thread</span><span class="o">(()</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="nc">Reference</span> <span class="n">remove</span> <span class="o">=</span> <span class="n">referenceQueue</span><span class="o">.</span><span class="na">remove</span><span class="o">();</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"remove:"</span> <span class="o">+</span> <span class="n">remove</span><span class="o">);</span> <span class="o">}).</span><span class="na">start</span><span class="o">();</span> <span class="k">new</span> <span class="nc">WeakReference</span><span class="o">&lt;&gt;(</span><span class="k">new</span> <span class="nc">String</span><span class="o">(</span><span class="s">"111"</span><span class="o">),</span> <span class="n">referenceQueue</span><span class="o">);</span> <span class="nc">System</span><span class="o">.</span><span class="na">gc</span><span class="o">();</span> <span class="c1">// 什么都不打印</span> </code></pre></div> </div> </li> <li>下例说明,<strong>显式调用 System.gc() 的线程,如果没有在该线程的任何地方使用到这个 ref,那么并不会触发 ref 的回收</strong>。想要 ref 能够被回收,那么需要在 gc 线程的任意一个位置出现 ref。 <div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">ReferenceQueue</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">&gt;</span> <span class="n">referenceQueue</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ReferenceQueue</span><span class="o">&lt;&gt;();</span> <span class="c1">// 监视线程</span> <span class="k">new</span> <span class="nf">Thread</span><span class="o">(()</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="nc">Reference</span> <span class="n">remove</span> <span class="o">=</span> <span class="n">referenceQueue</span><span class="o">.</span><span class="na">remove</span><span class="o">();</span> <span class="c1">// 不会捕获到引用的回收</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"remove:"</span> <span class="o">+</span> <span class="n">remove</span><span class="o">);</span> <span class="o">}).</span><span class="na">start</span><span class="o">();</span> <span class="nc">WeakReference</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">&gt;</span> <span class="n">ref</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">WeakReference</span><span class="o">&lt;&gt;(</span><span class="k">new</span> <span class="nc">String</span><span class="o">(</span><span class="s">"111"</span><span class="o">),</span> <span class="n">referenceQueue</span><span class="o">);</span> <span class="c1">// gc 线程</span> <span class="k">new</span> <span class="nf">Thread</span><span class="o">(()</span> <span class="o">-&gt;</span> <span class="o">{</span> <span class="c1">// Object a = ref; // 取消注释,则可以捕获到回收</span> <span class="nc">Thread</span><span class="o">.</span><span class="na">sleep</span><span class="o">(</span><span class="mi">1000</span><span class="o">);</span> <span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"gc"</span><span class="o">);</span> <span class="nc">System</span><span class="o">.</span><span class="na">gc</span><span class="o">();</span> <span class="c1">// 在另一个线程 System.gc()</span> <span class="o">}).</span><span class="na">start</span><span class="o">();</span> </code></pre></div> </div> </li> </ol> 2019-08-03T11:13:00+00:00 https://ysmull.cn/blog/reference.html https://ysmull.cn/blog/reference.html 如何加速你的 count 查询 <ul id="markdown-toc"> <li><a href="#id-背景" id="markdown-toc-id-背景">背景</a></li> <li><a href="#id-问题" id="markdown-toc-id-问题">问题</a></li> <li><a href="#id-解决" id="markdown-toc-id-解决">解决</a></li> <li><a href="#id-延伸" id="markdown-toc-id-延伸">延伸</a></li> </ul> <h2 id="id-背景">背景</h2> <p>项目中,经常会有一些统计类的查询,大概模式是:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="k">table_name</span> <span class="k">where</span> <span class="p">...</span> </code></pre></div></div> <p>如何加速此类查询,大部分人首先想到的办法可能就是,「根据 where 条件」给表加索引。<br /> 那么如果是整表 count 呢,还有提高速度的办法吗?答案是,有的。</p> <h2 id="id-问题">问题</h2> <p>线上有一张 news 表,会有爬虫不断的向该表插入爬取的新闻数据(所以下文中每次count的结果总不相同是因为数据确实增加了)</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">create</span> <span class="k">table</span> <span class="n">news</span> <span class="p">(</span> <span class="n">id</span> <span class="nb">int</span> <span class="n">auto_increment</span> <span class="k">primary</span> <span class="k">key</span><span class="p">,</span> <span class="k">type</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span> <span class="k">default</span> <span class="s1">''</span> <span class="k">not</span> <span class="k">null</span> <span class="k">comment</span> <span class="s1">'新闻分类'</span><span class="p">,</span> <span class="n">title</span> <span class="nb">text</span> <span class="k">null</span><span class="p">,</span> <span class="n">description</span> <span class="nb">text</span> <span class="k">null</span><span class="p">,</span> <span class="n">content</span> <span class="nb">mediumtext</span> <span class="k">null</span><span class="p">,</span> <span class="n">pics</span> <span class="n">json</span> <span class="k">null</span><span class="p">,</span> <span class="k">source</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">255</span><span class="p">)</span> <span class="k">null</span><span class="p">,</span> <span class="n">create_time</span> <span class="nb">timestamp</span> <span class="k">default</span> <span class="k">CURRENT_TIMESTAMP</span> <span class="k">null</span> <span class="p">)</span> <span class="k">collate</span> <span class="o">=</span> <span class="n">utf8mb4_unicode_ci</span><span class="p">;</span> </code></pre></div></div> <p>直接执行</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mysql</span><span class="o">&gt;</span> <span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="n">news</span><span class="p">;</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="mi">678532</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="mi">1</span> <span class="k">row</span> <span class="k">in</span> <span class="k">set</span> <span class="p">(</span><span class="mi">1</span> <span class="k">min</span> <span class="mi">3</span><span class="p">.</span><span class="mi">57</span> <span class="n">sec</span><span class="p">)</span> <span class="n">mysql</span><span class="o">&gt;</span> <span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="n">news</span><span class="p">;</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="mi">678538</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="mi">1</span> <span class="k">row</span> <span class="k">in</span> <span class="k">set</span> <span class="p">(</span><span class="mi">50</span><span class="p">.</span><span class="mi">18</span> <span class="n">sec</span><span class="p">)</span> </code></pre></div></div> <p>可以看到,执行该 SQL 非常的耗时,我们查看一下执行计划:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mysql</span><span class="o">&gt;</span> <span class="k">explain</span> <span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="n">news</span> <span class="err">\</span><span class="k">G</span><span class="p">;</span> <span class="o">***************************</span> <span class="mi">1</span><span class="p">.</span> <span class="k">row</span> <span class="o">***************************</span> <span class="n">id</span><span class="p">:</span> <span class="mi">1</span> <span class="n">select_type</span><span class="p">:</span> <span class="k">SIMPLE</span> <span class="k">table</span><span class="p">:</span> <span class="n">news</span> <span class="n">partitions</span><span class="p">:</span> <span class="k">NULL</span> <span class="k">type</span><span class="p">:</span> <span class="k">index</span> <span class="n">possible_keys</span><span class="p">:</span> <span class="k">NULL</span> <span class="k">key</span><span class="p">:</span> <span class="k">PRIMARY</span> <span class="n">key_len</span><span class="p">:</span> <span class="mi">4</span> <span class="k">ref</span><span class="p">:</span> <span class="k">NULL</span> <span class="k">rows</span><span class="p">:</span> <span class="mi">440632</span> <span class="n">filtered</span><span class="p">:</span> <span class="mi">100</span><span class="p">.</span><span class="mi">00</span> <span class="n">Extra</span><span class="p">:</span> <span class="k">Using</span> <span class="k">index</span> <span class="mi">1</span> <span class="k">row</span> <span class="k">in</span> <span class="k">set</span><span class="p">,</span> <span class="mi">1</span> <span class="n">warning</span> <span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span> <span class="n">sec</span><span class="p">)</span> </code></pre></div></div> <p>执行计划说,该查询使用了主键索引,由于是主键索引,该查询自然而然的也是覆盖索引,并且 <code class="language-plaintext highlighter-rouge">filtered</code> 也是 <code class="language-plaintext highlighter-rouge">100.00</code>。那么如何对这句SQL进行加速呢?</p> <h2 id="id-解决">解决</h2> <p>由于 count(*) 类型的查询不需要去任何一列的数据,主键索引(clustered index)的叶子节点存储了完整的行数据(上例中甚至还有很多的 blob 类字段,数据页都存储不下了),遍历主键索引会带来较大的磁盘 IO,所以解决办法就呼之欲出了———随便找个列,建一个小巧的辅助索引。</p> <p>我们选择 source 列来建索引。</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">create</span> <span class="k">index</span> <span class="n">news_source_index</span> <span class="k">on</span> <span class="n">news</span> <span class="p">(</span><span class="k">source</span><span class="p">);</span> </code></pre></div></div> <p>索引建好后,我们查看执行计划:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mysql</span><span class="o">&gt;</span> <span class="k">explain</span> <span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="n">news</span> <span class="err">\</span><span class="k">G</span><span class="p">;</span> <span class="o">***************************</span> <span class="mi">1</span><span class="p">.</span> <span class="k">row</span> <span class="o">***************************</span> <span class="n">id</span><span class="p">:</span> <span class="mi">1</span> <span class="n">select_type</span><span class="p">:</span> <span class="k">SIMPLE</span> <span class="k">table</span><span class="p">:</span> <span class="n">news</span> <span class="n">partitions</span><span class="p">:</span> <span class="k">NULL</span> <span class="k">type</span><span class="p">:</span> <span class="k">index</span> <span class="n">possible_keys</span><span class="p">:</span> <span class="k">NULL</span> <span class="k">key</span><span class="p">:</span> <span class="n">news_source_index</span> <span class="n">key_len</span><span class="p">:</span> <span class="mi">1023</span> <span class="k">ref</span><span class="p">:</span> <span class="k">NULL</span> <span class="k">rows</span><span class="p">:</span> <span class="mi">440697</span> <span class="n">filtered</span><span class="p">:</span> <span class="mi">100</span><span class="p">.</span><span class="mi">00</span> <span class="n">Extra</span><span class="p">:</span> <span class="k">Using</span> <span class="k">index</span> <span class="mi">1</span> <span class="k">row</span> <span class="k">in</span> <span class="k">set</span><span class="p">,</span> <span class="mi">1</span> <span class="n">warning</span> <span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">02</span> <span class="n">sec</span><span class="p">)</span> </code></pre></div></div> <p>根据 Extra 字段,你会发现,该查询仍然是覆盖索引,但是 InnoDB 选择了使用我们刚刚创建的辅助索引,这是因为,<strong>辅助索引远小于主键索引</strong>,遍历辅助索引的代价较小一些。再一次执行查询:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mysql</span><span class="o">&gt;</span> <span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="n">news</span><span class="p">;</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="mi">678129</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="mi">1</span> <span class="k">row</span> <span class="k">in</span> <span class="k">set</span> <span class="p">(</span><span class="mi">1</span><span class="p">.</span><span class="mi">32</span> <span class="n">sec</span><span class="p">)</span> <span class="n">mysql</span><span class="o">&gt;</span> <span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="n">news</span><span class="p">;</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="mi">678129</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="mi">1</span> <span class="k">row</span> <span class="k">in</span> <span class="k">set</span> <span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">09</span> <span class="n">sec</span><span class="p">)</span> </code></pre></div></div> <p>连续两次执行该SQL的时间分别从 <em><code class="language-plaintext highlighter-rouge">1 min 3.57 sec</code>、<code class="language-plaintext highlighter-rouge">50.18 sec</code></em> 变成了 <em><code class="language-plaintext highlighter-rouge">1.32 sec</code>、<code class="language-plaintext highlighter-rouge">0.09 sec</code></em></p> <h2 id="id-延伸">延伸</h2> <p>上文讲解的例子是一个没有 where 条件的 SQL,提高 <code class="language-plaintext highlighter-rouge">count(*)</code> 速度的方法是随便建一个代价较小的辅助索引。如果是带有查询条件的查询呢?我们是不是一定要建立精准命中查询条件的索引呢?答案是否定的。</p> <p><strong>对于 count 型查询,我们不一定需要一个完美切合查询条件的索引。</strong></p> <p>下面用一个例子来说明,首先还是举一个创建精确索引的例子。</p> <p>还是上面的 news 表,我们想执行一个范围查询:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mysql</span><span class="o">&gt;</span> <span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="n">news</span> <span class="k">where</span> <span class="n">create_time</span> <span class="o">&gt;</span> <span class="n">curdate</span><span class="p">();</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="mi">996</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="mi">1</span> <span class="k">row</span> <span class="k">in</span> <span class="k">set</span> <span class="p">(</span><span class="mi">1</span> <span class="k">min</span> <span class="mi">44</span><span class="p">.</span><span class="mi">76</span> <span class="n">sec</span><span class="p">)</span> </code></pre></div></div> <p>该查询用不到任何索引,所以速度非常慢,因此,我们首先想到的是,给 <code class="language-plaintext highlighter-rouge">create_time</code> 字段建立索引。</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mysql</span><span class="o">&gt;</span> <span class="k">create</span> <span class="k">index</span> <span class="n">news_create_time_index</span> <span class="k">on</span> <span class="n">news</span> <span class="p">(</span><span class="n">create_time</span><span class="p">);</span> <span class="n">Query</span> <span class="n">OK</span><span class="p">,</span> <span class="mi">0</span> <span class="k">rows</span> <span class="n">affected</span> <span class="p">(</span><span class="mi">1</span> <span class="k">min</span> <span class="mi">16</span><span class="p">.</span><span class="mi">94</span> <span class="n">sec</span><span class="p">)</span> <span class="n">Records</span><span class="p">:</span> <span class="mi">0</span> <span class="n">Duplicates</span><span class="p">:</span> <span class="mi">0</span> <span class="n">Warnings</span><span class="p">:</span> <span class="mi">0</span> </code></pre></div></div> <p>索引建好后,我们查看执行计划,可以用上这个索引,并且查询时间已经可以忽略不计了:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mysql</span><span class="o">&gt;</span> <span class="k">explain</span> <span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="n">news</span> <span class="k">where</span> <span class="n">create_time</span> <span class="o">&gt;</span> <span class="n">curdate</span><span class="p">()</span> <span class="err">\</span><span class="k">G</span><span class="p">;</span> <span class="o">***************************</span> <span class="mi">1</span><span class="p">.</span> <span class="k">row</span> <span class="o">***************************</span> <span class="n">id</span><span class="p">:</span> <span class="mi">1</span> <span class="n">select_type</span><span class="p">:</span> <span class="k">SIMPLE</span> <span class="k">table</span><span class="p">:</span> <span class="n">news</span> <span class="n">partitions</span><span class="p">:</span> <span class="k">NULL</span> <span class="k">type</span><span class="p">:</span> <span class="n">range</span> <span class="n">possible_keys</span><span class="p">:</span> <span class="n">news_create_time_index</span> <span class="k">key</span><span class="p">:</span> <span class="n">news_create_time_index</span> <span class="n">key_len</span><span class="p">:</span> <span class="mi">5</span> <span class="k">ref</span><span class="p">:</span> <span class="k">NULL</span> <span class="k">rows</span><span class="p">:</span> <span class="mi">1104</span> <span class="n">filtered</span><span class="p">:</span> <span class="mi">100</span><span class="p">.</span><span class="mi">00</span> <span class="n">Extra</span><span class="p">:</span> <span class="k">Using</span> <span class="k">where</span><span class="p">;</span> <span class="k">Using</span> <span class="k">index</span> <span class="mi">1</span> <span class="k">row</span> <span class="k">in</span> <span class="k">set</span><span class="p">,</span> <span class="mi">1</span> <span class="n">warning</span> <span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">00</span> <span class="n">sec</span><span class="p">)</span> <span class="n">mysql</span><span class="o">&gt;</span> <span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="n">news</span> <span class="k">where</span> <span class="n">create_time</span> <span class="o">&gt;</span> <span class="n">curdate</span><span class="p">();</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="mi">1104</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="mi">1</span> <span class="k">row</span> <span class="k">in</span> <span class="k">set</span> <span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">01</span> <span class="n">sec</span><span class="p">)</span> </code></pre></div></div> <p><strong><em>(注意下面的转折)</em></strong></p> <p>然而,假设如果这张表已经有了<strong>包含 create_time 的联合索引</strong>,并且 create_time <strong>不是联合索引的第一列</strong>,那么 InnoDB 可否使用那个联合索引来优化这个查询呢?</p> <p>比如假设我们已经有了(source, create_time) 的联合索引,仅仅看 where 条件,是对 create_time 进行范围查找,显然是用不上这个联合索引的,因为联合索引只有第一列是有序的,其它列只相对于前一列保持有序。下面我们来实验一下,看看实际情况如何。</p> <p>首先我们删除已经存在的 create_time 的索引,然后建立一个 (source, create_time) 的联合索引:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mysql</span><span class="o">&gt;</span> <span class="k">create</span> <span class="k">index</span> <span class="n">news_source_create_time_index</span> <span class="k">on</span> <span class="n">news</span> <span class="p">(</span><span class="k">source</span><span class="p">,</span> <span class="n">create_time</span><span class="p">);</span> <span class="n">Query</span> <span class="n">OK</span><span class="p">,</span> <span class="mi">0</span> <span class="k">rows</span> <span class="n">affected</span> <span class="p">(</span><span class="mi">40</span><span class="p">.</span><span class="mi">95</span> <span class="n">sec</span><span class="p">)</span> <span class="n">Records</span><span class="p">:</span> <span class="mi">0</span> <span class="n">Duplicates</span><span class="p">:</span> <span class="mi">0</span> <span class="n">Warnings</span><span class="p">:</span> <span class="mi">0</span> <span class="n">mysql</span><span class="o">&gt;</span> <span class="k">drop</span> <span class="k">index</span> <span class="n">news_create_time_index</span> <span class="k">on</span> <span class="n">news</span><span class="p">;</span> <span class="n">Query</span> <span class="n">OK</span><span class="p">,</span> <span class="mi">0</span> <span class="k">rows</span> <span class="n">affected</span> <span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">02</span> <span class="n">sec</span><span class="p">)</span> <span class="n">Records</span><span class="p">:</span> <span class="mi">0</span> <span class="n">Duplicates</span><span class="p">:</span> <span class="mi">0</span> <span class="n">Warnings</span><span class="p">:</span> <span class="mi">0</span> </code></pre></div></div> <p>(吃了个中饭回来)<br /> 然后我们看一看这条查询的执行计划:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mysql</span><span class="o">&gt;</span> <span class="k">explain</span> <span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="n">news</span> <span class="k">where</span> <span class="n">create_time</span> <span class="o">&gt;</span> <span class="n">curdate</span><span class="p">()</span> <span class="err">\</span><span class="k">G</span><span class="p">;</span> <span class="o">***************************</span> <span class="mi">1</span><span class="p">.</span> <span class="k">row</span> <span class="o">***************************</span> <span class="n">id</span><span class="p">:</span> <span class="mi">1</span> <span class="n">select_type</span><span class="p">:</span> <span class="k">SIMPLE</span> <span class="k">table</span><span class="p">:</span> <span class="n">news</span> <span class="n">partitions</span><span class="p">:</span> <span class="k">NULL</span> <span class="k">type</span><span class="p">:</span> <span class="k">index</span> <span class="n">possible_keys</span><span class="p">:</span> <span class="n">news_source_create_time_index</span> <span class="k">key</span><span class="p">:</span> <span class="n">news_source_create_time_index</span> <span class="n">key_len</span><span class="p">:</span> <span class="mi">1028</span> <span class="k">ref</span><span class="p">:</span> <span class="k">NULL</span> <span class="k">rows</span><span class="p">:</span> <span class="mi">441733</span> <span class="n">filtered</span><span class="p">:</span> <span class="mi">33</span><span class="p">.</span><span class="mi">33</span> <span class="n">Extra</span><span class="p">:</span> <span class="k">Using</span> <span class="k">where</span><span class="p">;</span> <span class="k">Using</span> <span class="k">index</span> <span class="mi">1</span> <span class="k">row</span> <span class="k">in</span> <span class="k">set</span><span class="p">,</span> <span class="mi">1</span> <span class="n">warning</span> <span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">03</span> <span class="n">sec</span><span class="p">)</span> </code></pre></div></div> <p>根据执行计划我们发现,对 create_time 的范围查询,竟然可以用到 (source, create_time) 的联合索引,实际执行该查询:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mysql</span><span class="o">&gt;</span> <span class="k">select</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="k">from</span> <span class="n">news</span> <span class="k">where</span> <span class="n">create_time</span> <span class="o">&gt;</span> <span class="n">curdate</span><span class="p">();</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="o">|</span> <span class="mi">1173</span> <span class="o">|</span> <span class="o">+</span><span class="c1">----------+</span> <span class="mi">1</span> <span class="k">row</span> <span class="k">in</span> <span class="k">set</span> <span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="mi">27</span> <span class="n">sec</span><span class="p">)</span> </code></pre></div></div> <p>查询仍然非常的高效,这是为什么呢?</p> <p>原因在于,尽管 create_time 是无序的,但是对于 count(*) 类型的查询,我们需要使用到的字段其实只有 create_time 而已,而辅助索引 (source, create_time) 的 B+ 树中,已经包含了所有的 create_time 信息,可以「覆盖」我们的查询条件,足够我们对「记录数」进行统计,因此,不需要去遍历巨大的主键索引的 B+ 树。然而,由于 create_time 是无序的,我们仍然需要对所有的索引进行排序之后再筛选出满足条件的记录,所以 Extra 中包含了 <code class="language-plaintext highlighter-rouge">Using where</code>,但这比直接在主键索引中进行筛选,已经减少了很多的磁盘IO,所以仍然保持了查询性能。</p> 2019-07-28T00:00:00+00:00 https://ysmull.cn/blog/count.html https://ysmull.cn/blog/count.html InnoDB存储引擎之 —— 索引算法(一)(草稿...) <p>InnoDB 存储引擎表示索引组织表,表中数据按照主键顺序存放。<br /> 索引应该在一开始开发的时候就进行添加。<br /> 索引太少,影响性能。索引太多,也影响性能,比如太多索引会导致磁盘使用率非常高。</p> <p>索引的类型</p> <ol> <li>B+ 树索引<br /> 传统意义上的索引,最常用,最有效。<br /> B+ 树索引不能定位到具体的行,只能定位到具体的页,然后把页读到内存中查找满足条件的行。</li> <li>全文索引</li> <li>哈希索引<br /> InnoDB 的哈希索引是自适应的,会根据表的使用情况自动为表生成,不能认为干预。</li> </ol> <p>背景知识:<br /> 二分查找:<br /> 第一个二分查找在1946年出现,第一个正确的二分查找法在1962年才出现。</p> <p>B+树:<br /> 省略</p> <p>B+树索引:</p> <ol> <li> <p>聚集索引(clustered index)<br /> 根据每张表的主键构建的一颗B+树(一张表只能有一个聚集索引),叶子节点存放的是一整行的信息。聚集索引的叶子也称作<strong>数据页</strong>,每个数据页通过双向链表链接。</p> </li> <li> <p>辅助索引(secondary index)<br /> 叶子节点存放的<strong>不是</strong>一整行的信息。</p> </li> </ol> 2019-07-27T00:00:00+00:00 https://ysmull.cn/blog/key1.html https://ysmull.cn/blog/key1.html InnoDB存储引擎之 —— 锁(二) <ul id="markdown-toc"> <li><a href="#id-锁的算法" id="markdown-toc-id-锁的算法">锁的算法</a> <ul> <li><a href="#id-锁的例子" id="markdown-toc-id-锁的例子">锁的例子</a></li> </ul> </li> <li><a href="#id-阻塞超时" id="markdown-toc-id-阻塞超时">阻塞&amp;超时</a></li> <li><a href="#id-死锁" id="markdown-toc-id-死锁">死锁</a> <ul> <li><a href="#id-死锁例子" id="markdown-toc-id-死锁例子">死锁例子</a></li> </ul> </li> <li><a href="#id-锁升级" id="markdown-toc-id-锁升级">锁升级</a></li> </ul> <h2 id="id-锁的算法">锁的算法</h2> <ol> <li> <p>Record Lock<br /> 只锁住行。总是去锁定索引,如果没有建索引,就锁主键。</p> </li> <li> <p>Gap Lock<br /> 锁住一个范围,不包含记录本身</p> </li> <li> <p>Next-Key Lock (Previous-Key Lock)<br /> 锁住一个范围,包含记录本身。</p> </li> </ol> <p>假设一个索引有10,11,13,20这四个值。那么有如下五个区间:</p> <div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(-∞, 10) (10, 11) (11, 13) (13, 20) (20, +∞) </code></pre></div></div> <ul> <li><code class="language-plaintext highlighter-rouge">Gap Lock</code> 锁的就是上述区间。</li> <li><code class="language-plaintext highlighter-rouge">Next-Key Locking</code> 锁的是上述区间的左开右闭区间。</li> <li><code class="language-plaintext highlighter-rouge">Previous-Key Locking</code> 锁的是上述区间的左闭右开区间。</li> </ul> <p>InnoDB 加锁有如下三条特殊规则:</p> <ol> <li><strong>当查询的索引仅含有唯一索引的时候,<code class="language-plaintext highlighter-rouge">Next-Key Lock</code> 会降级为 <code class="language-plaintext highlighter-rouge">Record Lock</code>。</strong>(联合唯一索引,每一个索引列都要查)</li> <li><strong>InnoDB 还会对锁住的辅助索引加 <code class="language-plaintext highlighter-rouge">Next-Key Lock</code>,并且会给下一个键值加 <code class="language-plaintext highlighter-rouge">Gap Lock</code></strong></li> <li><strong>插入某记录时候,会检查插入记录的下一条记录是否被锁住了,如果锁住了,则不允许插入(阻塞)。</strong></li> </ol> <p>InnoDB 在默认的 REPETABLE READ 隔离级别下,使用 Next-Key Lock,在 READ COMMITED 隔离级别下,仅使用 Record Lock</p> <h3 id="id-锁的例子">锁的例子</h3> <p>下面举几个例子来讲解:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">CREATE</span> <span class="k">TABLE</span> <span class="n">z</span> <span class="p">(</span><span class="n">a</span> <span class="nb">INT</span><span class="p">,</span> <span class="n">b</span><span class="p">,</span> <span class="nb">INT</span><span class="p">,</span> <span class="k">PRIMARY</span> <span class="k">KEY</span><span class="p">(</span><span class="n">a</span><span class="p">),</span> <span class="k">KEY</span><span class="p">(</span><span class="n">b</span><span class="p">))</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">z</span> <span class="k">SELECT</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">z</span> <span class="k">SELECT</span> <span class="mi">3</span><span class="p">,</span> <span class="mi">1</span><span class="p">;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">z</span> <span class="k">SELECT</span> <span class="mi">5</span><span class="p">,</span> <span class="mi">3</span><span class="p">;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">z</span> <span class="k">SELECT</span> <span class="mi">7</span><span class="p">,</span> <span class="mi">6</span><span class="p">;</span> <span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">z</span> <span class="k">SELECT</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">8</span> </code></pre></div></div> <p><strong>会话A</strong>:执行</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">z</span> <span class="k">WHERE</span> <span class="n">b</span><span class="o">=</span><span class="mi">3</span> <span class="k">FOR</span> <span class="k">UPDATE</span><span class="p">;</span> </code></pre></div></div> <p>我们来分析会锁住哪些记录:<br /> a 是聚集索引,加 Record Lock<br /> b 是辅助索引,加 Next-Key Lock,下一个键值加 Gap Lock</p> <p>索引 a 被锁住的记录为:</p> <ul> <li>5 — Record Lock</li> </ul> <p>索引 b 被锁住的记录为:</p> <ul> <li>(1, 3] — Next-Key Lock</li> <li>(3, 6) — Gap Lock</li> </ul> <p><strong>会话B</strong>:如果执行下面这些 SQL ,则都会被阻塞:</p> <ul> <li> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="k">FROM</span> <span class="n">z</span> <span class="k">WHERE</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">5</span> <span class="k">LOCK</span> <span class="k">IN</span> <span class="k">SHARE</span> <span class="k">MODE</span><span class="p">;</span> </code></pre></div> </div> <p><strong>原因:</strong>a = 5 的索引已经被加了 Record Lock 的排它锁,所以无法再加一个共享锁了。</p> </li> <li> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">z</span> <span class="k">SELECT</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">2</span><span class="p">;</span> </code></pre></div> </div> <p><strong>原因:</strong> 2 落在区间 <code class="language-plaintext highlighter-rouge">(1, 3]</code> Next-Key Lock 中</p> </li> <li> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">z</span> <span class="k">SELECT</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">5</span><span class="p">;</span> </code></pre></div> </div> <p><strong>原因:</strong> 5 落在区间 <code class="language-plaintext highlighter-rouge">(3, 6)</code> Gap Lock 中</p> </li> <li> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">INSERT</span> <span class="k">INTO</span> <span class="n">z</span> <span class="k">SELECT</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">2</span><span class="p">;</span> </code></pre></div> </div> <p><strong>原因:</strong> 欲插入 b = 2 的记录,而 b = 3 的记录已经被锁住了。因此插入被阻塞。插入 b = 0 的记录没有问题,因为 b = 1 没有被锁。(<em>实践发现这里 b 等于 1 的记录也可以插入,但是 b = 2 的记录被 next-key lock 锁住了呀,为什么呢?</em>)</p> </li> </ul> <p>实践了一些其它例子:</p> <ul> <li> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">select</span> <span class="o">*</span> <span class="k">from</span> <span class="n">z</span> <span class="k">where</span> <span class="n">b</span> <span class="o">&gt;</span> <span class="mi">3</span><span class="p">;</span> </code></pre></div> </div> <p>不会锁住 a = 5 的记录。同时 b 的 next-key lock 区间为 (3, +∞)<br /> 可以给 b = 3 的记录加 S 锁 : <code class="language-plaintext highlighter-rouge">select * from z where b = 3 lock in share mode;</code><br /> 不可以 insert b = 3 的记录: <code class="language-plaintext highlighter-rouge">INSERT INTO z SELECT 2, 3;</code></p> </li> <li> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">select</span> <span class="o">*</span> <span class="k">from</span> <span class="n">z</span> <span class="k">where</span> <span class="n">b</span> <span class="o">&gt;=</span> <span class="mi">3</span><span class="p">;</span> </code></pre></div> </div> <p>会锁住 a = 5 的记录。同时 b 的 next-key lock 区间为 [3, +∞)<br /> 不可以给 b = 3 的记录加 S 锁 : <code class="language-plaintext highlighter-rouge">select * from z where b = 3 lock in share mode;</code><br /> 不可以 insert b = 3 的记录: <code class="language-plaintext highlighter-rouge">INSERT INTO z SELECT 2, 3;</code></p> </li> </ul> <h2 id="id-阻塞超时">阻塞&amp;超时</h2> <p><strong>错误代码:</strong> 1205</p> <ul> <li><strong>innodb_lock_wait_timeout</strong>: 用来控制等待的时间(默认是 50 秒),<strong>可以在 MYSQL 数据库运行时进行调整</strong>。</li> <li><strong>innodb_rollback_on_timeout</strong>: 用来设定是否在等待超时时对进行中的事务进行回滚操作(默认是 OFF,代表不回滚)。<strong>不可在启动后进行修改。</strong></li> </ul> <p>当发生超时,MYSQL 数据库会抛出一个 1205 的错误,事务出现异常,在大多数情况下不会自动回滚,需要应用层自己去控制是commit还是rollback。</p> <h2 id="id-死锁">死锁</h2> <p>多个事务因争夺锁资源,造成相互等待,若无外力作用,无法推进下去。<br /> <strong>错误代码:</strong> 1213<br /> 两种方案:</p> <ol> <li>超时机制<br /> FIFO。谁先等待,谁先回滚。</li> <li>wait-for graph<br /> <strong>主动监测死锁,回滚 undo 代价最小的事务</strong>。<br /> 具体实现细节需要看源码,可能是通过如下两个数据结构去构造有向图: <ul> <li>Transaction Wait Lists</li> <li>Lock Lists<br /> <img src="/img/2019-07-25-100908.png" alt="" /><br /> 死锁检测通常采用深度优先的算法实现,在 INNODB1.2 版本之前,都是采用递归方式实现。而从 1.2 版本开始,对 wait- for graph 的死锁检测进行了优化,用非递归的方式实现。</li> </ul> </li> </ol> <h3 id="id-死锁例子">死锁例子</h3> <ul> <li>例子一<br /> <img src="/img/2019-07-25-101419.png" alt="" /></li> <li>例子二<br /> <img src="/img/2019-07-25-101438.png" alt="" /></li> </ul> <h2 id="id-锁升级">锁升级</h2> <blockquote> <p>锁升级(Lock Escalation)是指将当前锁的粒度降低。举例来说,数据库可以把个表的 1000 个行锁升级为一个页锁,或者将页锁升级为表锁。如果在数据库的设计中认为锁是一种稀有资源,而且想避免锁的开销,那数据库中会频繁出现锁升级现象。<br /> Microsoft SQL Server 数据库的设计认为锁是一种稀有的资源,在适合的时候会自动地将行、键或分页锁升级为更粗粒度的表级锁。这种升级保护了系统资源,防止系统使用太多的内存来维护锁,在一定程度上提高了效率。<br /> 即使在 Microsoft SQL Server20 版本之后,SQL Server 数据库支持了行锁,但是其设计和 INNODB 存储引擎完全不同,在以下情况下依然可能发生锁升级:</p> <ol> <li>由一句单独的 SQL 语句在一个对象上持有的锁的数量超过了阈值,默认这个阈值为 5000。值得注意的是,如果是不同对象,则不会发生锁升级口</li> <li>锁资源占用的内存超过了激活内存的 40%时就会发生锁升级</li> </ol> <p>在 Microsoft SQL Server 数据库中,由于锁是一种稀有的资源,因此锁升级会带来一定的效率提高。但是锁升级带来的一个问题却是因为锁粒度的降低而导致并发性能的降低。</p> <p>INNODB 存储引擎不存在锁升级的问题。因为其不是根据每个记录来产生行锁的,相反,其根据每个事务访问的每个页对锁进行管理的,采用的是位图的方式。因此不管个事务锁住页中一个记录还是多个记录,其开销通常都是一致的。</p> </blockquote> 2019-07-25T00:00:00+00:00 https://ysmull.cn/blog/lock2.html https://ysmull.cn/blog/lock2.html 项目中遇到的锁与数据一致性问题 <ul id="markdown-toc"> <li><a href="#id-背景" id="markdown-toc-id-背景">背景</a></li> <li><a href="#id-问题" id="markdown-toc-id-问题">问题</a></li> <li><a href="#id-解决方案" id="markdown-toc-id-解决方案">解决方案</a> <ul> <li><a href="#id-方案一-外键约束" id="markdown-toc-id-方案一-外键约束">方案一: 外键约束</a></li> <li><a href="#id-方案二-一致性锁定读" id="markdown-toc-id-方案二-一致性锁定读">方案二: 一致性锁定读</a> <ul> <li><a href="#id-方案二的证明" id="markdown-toc-id-方案二的证明">方案二的证明</a></li> <li><a href="#id-方案二死锁举例" id="markdown-toc-id-方案二死锁举例">方案二死锁举例</a></li> </ul> </li> </ul> </li> </ul> <h2 id="id-背景">背景</h2> <p>项目表: project(id, name)</p> <p>报告表: report(id, project_id, name)</p> <p>有两个操作:</p> <ol> <li>操作一:删除项目 A (id = 1) <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">delete</span> <span class="k">from</span> <span class="n">project</span> <span class="k">where</span> <span class="n">id</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="k">delete</span> <span class="k">from</span> <span class="n">report</span> <span class="k">where</span> <span class="n">project_id</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> </code></pre></div> </div> </li> <li>操作二:给项目 A 新增一个报告 <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">insert</span> <span class="k">into</span> <span class="n">report</span><span class="p">(</span><span class="nv">`project_id`</span><span class="p">,</span> <span class="nv">`name`</span><span class="p">)</span> <span class="k">values</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'报告1'</span><span class="p">);</span> </code></pre></div> </div> </li> </ol> <h2 id="id-问题">问题</h2> <p>如何保证两个操作并行进行之后的数据一致性,即:操作一和操作二都结束后,数据库里没有项目 ID = 1 的报告。</p> <h2 id="id-解决方案">解决方案</h2> <h3 id="id-方案一-外键约束">方案一: 外键约束</h3> <p><strong>给 report 表的 project_id 字段加外键。</strong><br /> 在有外键之后,在插入 report 的时候,会尝试获取 project 表的相关记录的 X 锁。</p> <ol> <li>如果此时可以获取锁,那么此时操作一一定还没有执行删除项目操作,报告会添加成功。随后操作一在删除项目的时候,会把该报告删除。</li> <li>如果此时不能获取锁,说明操作一正在删除该插入报告所在的项目。等操作一的事务提交后,操作二会插入失败,报外键错误。</li> </ol> <p>注意,此时操作一所在的事务,SQL一定得按如下顺序执行,先删除依赖,否则报外键错误</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">delete</span> <span class="k">from</span> <span class="n">report</span> <span class="k">where</span> <span class="n">project_id</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> <span class="k">delete</span> <span class="k">from</span> <span class="n">project</span> <span class="k">where</span> <span class="n">id</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span> </code></pre></div></div> <h3 id="id-方案二-一致性锁定读">方案二: 一致性锁定读</h3> <p>操作二使用如下逻辑操作,在插入报告之前,先判断是否存在该项目,查询时给项目加上 S 锁(也可以 for update 加上 X 锁)。我们用存储过程描述这个过程:</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">DROP</span> <span class="k">PROCEDURE</span> <span class="n">IF</span> <span class="k">EXISTS</span> <span class="n">insert_report</span><span class="p">;</span> <span class="k">CREATE</span> <span class="k">procedure</span> <span class="n">insert_report</span><span class="p">(</span><span class="k">IN</span> <span class="n">pid</span> <span class="nb">int</span><span class="p">,</span> <span class="k">IN</span> <span class="n">title</span> <span class="nb">varchar</span><span class="p">(</span><span class="mi">255</span><span class="p">))</span> <span class="k">BEGIN</span> <span class="k">start</span> <span class="n">transaction</span> <span class="p">;</span> <span class="k">SELECT</span> <span class="n">id</span> <span class="k">into</span> <span class="o">@</span><span class="n">project_id</span> <span class="k">FROM</span> <span class="n">project</span> <span class="k">WHERE</span> <span class="n">id</span> <span class="o">=</span> <span class="mi">1</span> <span class="k">lock</span> <span class="k">in</span> <span class="k">share</span> <span class="k">mode</span><span class="p">;</span> <span class="n">if</span> <span class="o">@</span><span class="n">project_id</span> <span class="k">is</span> <span class="k">not</span> <span class="k">null</span> <span class="k">then</span> <span class="k">insert</span> <span class="k">into</span> <span class="n">new_report</span><span class="p">(</span><span class="nv">`project_id`</span><span class="p">,</span> <span class="nv">`name`</span><span class="p">)</span> <span class="n">value</span> <span class="p">(</span><span class="n">pid</span><span class="p">,</span> <span class="n">title</span><span class="p">);</span> <span class="k">end</span> <span class="n">if</span><span class="p">;</span> <span class="k">commit</span><span class="p">;</span> <span class="k">END</span><span class="p">;</span> <span class="k">call</span> <span class="n">insert_report</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s1">'报告1'</span><span class="p">);</span> </code></pre></div></div> <p>事务一的两句SQL语句还可以交换顺序,两个事务的操作排列组合之后非常多,我们怎么知道在任何执行顺序下,这样可以解决数据一致性问题呢?</p> <h4 id="id-方案二的证明">方案二的证明</h4> <p><strong>证明:</strong><br /> 当事务二执行 select id from project where id = 1 lock in share mode 返回结果后,到事务二commit之前的这段时间之内</p> <p>事务一要么「<strong>已经</strong>删除」了 project 表中 id = 1 的记录,要么还在「<strong>等待</strong>删除」 project 表中 id = 1 的记录。</p> <ol> <li> <p>事务一「已经删除」 project 表中 id 为 1 的记录<br /> 那么事务二不会执行 insert 语句添加 project_id 为 1 的 记录,最终保持了数据一致。</p> </li> <li> <p>事务一在「等待删除」project 表中 id 为 1 的记录,那么事务二接下来将插入 project_id=1 的 report</p> <ul> <li>2.1 <em>如果事务一已经删除了 project_id=1 的report</em><br /> 那么事务二无法插入 project_id=1 的 report ,导致死锁,触发 MySQL 回滚。</li> <li>2.2 <em>如果事务一还没有删除 project_id=1 的report</em><br /> 那么事务二可以插入 project_id=1 的 report ,并且可以commit,事务二comimt后,会释放 project 表中 id = 1 的记录的 S 锁,从而事务一可以拿到 project 中 id = 1 的记录的 X 锁,删除 project 表中 id= 1 的记录,然后删除 report 表中所有 project_id = 1 的记录。</li> </ul> <p>无论是 2.1 还是 2.2,最终数据库中没有 id=1 的 project,也没有 project_id=1 的 report,保持了一致性。</p> </li> </ol> <h4 id="id-方案二死锁举例">方案二死锁举例</h4> <p>上面的证明提到了可能会带来死锁,下面给出死锁的情况。<br /> 考虑如下的操作序列</p> <table> <thead> <tr> <th>事务1</th> <th>事务2</th> </tr> </thead> <tbody> <tr> <td><code class="language-plaintext highlighter-rouge">delete from report where project_id = 1; -- 给 report 表加了 X 锁</code></td> <td> </td> </tr> <tr> <td> </td> <td>SELECT id into @project_id FROM project WHERE id = 1 lock in share mode; – 给 project 表加了 S 锁</td> </tr> <tr> <td>delete from project where id = 1; – 尝试获取获取 project 表的 X 锁,<strong>阻塞</strong></td> <td> </td> </tr> <tr> <td> </td> <td>insert into report(project_id, name) value (1, ‘报告1’); –尝试获取 report 表的 X 锁,<strong>阻塞</strong></td> </tr> </tbody> </table> <p>这样就导致了死锁。还好 MySQL 会自动回滚其中一个事务,算是保证了数据一致性。</p> 2019-07-25T00:00:00+00:00 https://ysmull.cn/blog/lock-case.html https://ysmull.cn/blog/lock-case.html 记录一次数据库迁移 <ul id="markdown-toc"> <li><a href="#id-前情提要" id="markdown-toc-id-前情提要">前情提要</a></li> <li><a href="#id-问题发现" id="markdown-toc-id-问题发现">问题发现</a></li> <li><a href="#id-问题解决" id="markdown-toc-id-问题解决">问题解决</a></li> </ul> <h3 id="id-前情提要">前情提要</h3> <p>几个月前帮同学做了个新闻app,后端基本是(Spring Boot + Kotlin)技术栈,「后台管理」的前端用 react ,打包到 spring boot 的 jar 包里一起启动。服务器购买的华为云。</p> <p>整个产品由以下几个模块组成:</p> <ol> <li>爬虫(爬新闻,插入 MySQL 数据库)</li> <li>后台管理(多用户登陆、用户管理、广告管理、报表展示)</li> <li>新闻后端(给安卓客户端提供后端服务)</li> <li>安卓客户端(请另一个同学写的)</li> </ol> <p>同时我使用了<del>给我发工资的</del>优秀的<a href="https://youdata.163.com/">网易有数</a>来监控爬虫情况,每天发送爬虫日报。</p> <h3 id="id-问题发现">问题发现</h3> <p>周一发现,连续三天新闻量没有增加。<br /> <img src="/img/2019-07-23-075249.jpg" alt="" /></p> <p>登陆服务器后发现爬虫服务的日志如下:<br /> <img src="/img/2019-07-23-075250.jpg" alt="" /></p> <p>使用 <code class="language-plaintext highlighter-rouge">df -h</code> 命令查看,发现硬盘已经满了(云服务默认40G硬盘)。</p> <h3 id="id-问题解决">问题解决</h3> <p>首先,想着去数据库上删除一些记录来节省空间。结果发现,MySQL 只能执行 SELECT 语句,无法执行 DELETE 语句,哪怕一条记录也删不掉。<strong>(当磁盘满了之后,为什么delete语句也无法执行,有意思的问题)</strong></p> <p>存储新闻的 news 表有十几G,小水管根本备份不下来。删也删不掉,下载也下载不下来,数据又不能全部丢掉,因此只能考虑给硬盘扩容了。</p> <p>华为云上可以在不重启的情况下,给已有的云硬盘扩容。我先花了十几块钱,买了一个 5 GB 的扩容包,发现果然所有服务都恢复正常了(可以正常插入数据)。</p> <p>然后本着折腾的心态,申请了经费,又买了一块单独的 100 GB 云硬盘,决定用这个空白的 100 G 硬盘专门存放 MySQL 数据,防止数据过几天又满了的问题,同事等数据再多些,准备上 ELK 技术栈玩玩。</p> <p><img src="/img/2019-07-23-080933.png" alt="" /><br /> 购买后,使用 <code class="language-plaintext highlighter-rouge">fdisk -l</code> 看到已经有新硬盘挂载上来了:<br /> <img src="/img/2019-07-23-075251.jpg" alt="" /></p> <p>依次使用如下的命令初始化硬盘,并挂载硬盘到指定目录:</p> <ol> <li>使用 <code class="language-plaintext highlighter-rouge">fdisk /dev/vdb</code> 格式化新硬盘</li> <li>使用 <code class="language-plaintext highlighter-rouge">fmkfs.ext4 /dev/vdb</code> 创建分区</li> <li>使用 <code class="language-plaintext highlighter-rouge">mount /dev/vdb /root/dbDisk</code> 挂载硬盘到指定目录(/root/dbDisk)</li> </ol> <p>接下来,我们删除 MySQL 容器,把所有 MySQL 的数据文件移动到新目录。修改 docker-compose.yml 文件的 volumes 字段,使用 <code class="language-plaintext highlighter-rouge">docker-compose up -d</code> 命令启动 MySQL。</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3"</span> <span class="na">services</span><span class="pi">:</span> <span class="na">db</span><span class="pi">:</span> <span class="na">image</span><span class="pi">:</span> <span class="s2">"</span><span class="s">mysql:8.0.16"</span> <span class="na">restart</span><span class="pi">:</span> <span class="s">always</span> <span class="na">command</span><span class="pi">:</span> <span class="s">--max_allowed_packet=20971520 --default-authentication-plugin=mysql_native_password</span> <span class="na">environment</span><span class="pi">:</span> <span class="na">MYSQL_ROOT_PASSWORD</span><span class="pi">:</span> <span class="s1">'</span><span class="s">'</span><span class="err">****************</span><span class="s1">'</span><span class="s">'</span> <span class="na">MYSQL_USER</span><span class="pi">:</span> <span class="s1">'</span><span class="s">'</span><span class="err">****************</span><span class="s1">'</span><span class="s">'</span> <span class="na">MYSQL_PASSWORD</span><span class="pi">:</span> <span class="s1">'</span><span class="s">****************'</span> <span class="na">MYSQL_DATABASE</span><span class="pi">:</span> <span class="s1">'</span><span class="s">news'</span> <span class="na">MYSQL_ROOT_HOST</span><span class="pi">:</span> <span class="s1">'</span><span class="s">%'</span> <span class="na">TZ</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Asia/Shanghai'</span> <span class="na">volumes</span><span class="pi">:</span> <span class="pi">-</span> <span class="s">/root/dbDisk/dbData:/var/lib/mysql</span> <span class="na">ports</span><span class="pi">:</span> <span class="pi">-</span> <span class="s1">'</span><span class="s">4396:3306'</span> </code></pre></div></div> <p>至此,MySQL 数据迁移结束。</p> 2019-07-23T00:00:00+00:00 https://ysmull.cn/blog/mysql-migrate.html https://ysmull.cn/blog/mysql-migrate.html InnoDB存储引擎之 —— 锁(一) <ul id="markdown-toc"> <li><a href="#id-sql-server" id="markdown-toc-id-sql-server">SQL Server</a></li> <li><a href="#id-myisam" id="markdown-toc-id-myisam">MyISAM</a></li> <li><a href="#id-innodb" id="markdown-toc-id-innodb">InnoDB</a> <ul> <li><a href="#id-innodb-锁的类型" id="markdown-toc-id-innodb-锁的类型">InnoDB 锁的类型</a></li> <li><a href="#id-几张查看锁和事务的表" id="markdown-toc-id-几张查看锁和事务的表">几张查看锁和事务的表</a></li> <li><a href="#id-一致性非锁定读" id="markdown-toc-id-一致性非锁定读">一致性非锁定读</a></li> <li><a href="#id-一致性锁定读" id="markdown-toc-id-一致性锁定读">一致性锁定读</a></li> <li><a href="#id-自增锁2种实现方式" id="markdown-toc-id-自增锁2种实现方式">自增锁(2种实现方式)</a> <ul> <li><a href="#id-auto--inc-locking特殊的表锁" id="markdown-toc-id-auto--inc-locking特殊的表锁">AUTO- INC Locking(特殊的表锁)</a></li> <li><a href="#id-互斥量" id="markdown-toc-id-互斥量">互斥量</a></li> </ul> </li> <li><a href="#id-外键与锁" id="markdown-toc-id-外键与锁">外键与锁</a></li> </ul> </li> </ul> <h2 id="id-sql-server">SQL Server</h2> <p>2005 之前是页锁,页锁容易实现,并发写入性能比 MyISAM 要好。但是对于热点数据更新还是无能为力。</p> <p>2005 之后是<code class="language-plaintext highlighter-rouge">乐观并发</code>和<code class="language-plaintext highlighter-rouge">悲观并发</code>,乐观并发下支持行级锁,但与 InnoDB 的实现方式完全不同。在 SQL Server 下,锁是一种稀缺资源,锁越多开销越大,因此会导致锁升级,行锁会升级到表锁。</p> <h2 id="id-myisam">MyISAM</h2> <p>MyISAM 引擎是表锁设计,并发读性能没问题,并发写性能有问题,若插入在“底部”,还能有一定的并发写入性能。</p> <h2 id="id-innodb">InnoDB</h2> <p>InnoDB 锁的实现与 Oracle 比较像,提供<code class="language-plaintext highlighter-rouge">一致性非锁定读</code>和<code class="language-plaintext highlighter-rouge">行级锁</code>,行级锁没有额外开销,支持并发性和一致性。</p> <h3 id="id-innodb-锁的类型">InnoDB 锁的类型</h3> <ol> <li>共享锁(S Lock)<br /> 允许事务读一行数据</li> <li>排它锁(X Lock)<br /> 允许事务删除或更改一条数据</li> <li>意向共享锁(IS Lock)<br /> 事务想要获取一张表中某几行的共享锁</li> <li>意向排它锁(IX Lock)<br /> 事务想要获取一张表中某几行的排它锁</li> <li>自增锁</li> </ol> <p><img src="/img/2019-07-20-074233.png" alt="" /><br /> 例如,如果一个表上已经有了S表锁,那么尝试修改该表的几行记录,是不可以的,因为IX锁与S锁是不兼容的。</p> <h3 id="id-几张查看锁和事务的表">几张查看锁和事务的表</h3> <ol> <li>INNODB_TRX<br /> 可以查看事务的状态<br /> <img src="/img/2019-07-20-074836.png" alt="" /></li> <li>INNODB_LOCKS<br /> 可以查看哪些表哪些记录被加了哪些锁。<br /> <img src="/img/2019-07-20-074941.png" alt="" /></li> <li>INNODB_LOCK_WAITS<br /> 已经有了前面两张表,判断事务的等待情况,<strong>可以很清楚的看到一个事务被哪个事务和锁阻塞</strong><br /> <img src="/img/2019-07-20-075100.png" alt="" /></li> </ol> <h3 id="id-一致性非锁定读">一致性非锁定读</h3> <p>InnoDB 存储引擎通过多版本控制(multi versioning)的方式来读取当前执行时间数据库中行的数据。如果读取的行正在执行 DELETE 或 UPDATE 操作,这时读取操作不会因此去等待行上锁的释放。而是去读取行的一个快照数据,如图所示:<br /> <img src="/img/2019-07-20-075600.png" alt="" /></p> <p>快照数据是指该行的之前版本的数据,该实现是通过 undo 段来完成。而 undo 用来在事务中回滚数据,因此快照数据本身是没有额外的开销。此外,读取快照数据是不需要上锁的,因为没有事务需要对历史的数据进行修改操作。</p> <p>快照数据其实就是当前行数据之前的历史版本,每行记录可能有多个版本。一个行记录可能有不止一个快照数据,一般称这种技术为行多版本技术。由此带来的并发控制,称之为<strong>多版本并发控制(Multi Version Concurrency Control, MVCC)</strong></p> <p><strong>非锁定读机制极大地提高了数据库的并发性</strong>。在事务隔离级别 READ COMMITTED 和 REPEATABLE READ (INNODB 存储引擎的默认事务隔离级别)下,INNODB 存储引擎使用非锁定的一致性读。然而,对于快照数据的定义却不相同。对于 READ COMMITTED 的事务隔离级别,它总是读取行的最新版本,如果行被锁定了,则读取该行版本的最新一个快照(<strong>也就是当前事务自己建立的那个快照</strong>)。而对于 REPEATABLE READ 的事务隔离级别,总是读取事务开始时的行数据(<em>这里应该也包含自己建立的快照数据</em>)。</p> <h3 id="id-一致性锁定读">一致性锁定读</h3> <p>如下两种语句必须在事务中使用,当事务提交了,锁也就释放了。</p> <ul> <li><strong>SELECT … FOR UPDATE</strong><br /> 对读取的行记录加一个 X 锁,其他事务不能对已锁定的行加上任何锁。<br /> <strong>即使读取的行已被执行了 SELECT… FOR UPDATE,还是可以进行一致性非锁定读的。</strong></li> <li><strong>SELECT … LOCK IN SHARE MODE</strong><br /> 对读取的行记录加一个 S 锁,其他事务可以向被锁定的行加 S 锁,但是如果加 X 锁,则会被阻塞。</li> </ul> <h3 id="id-自增锁2种实现方式">自增锁(2种实现方式)</h3> <h4 id="id-auto--inc-locking特殊的表锁">AUTO- INC Locking(特殊的表锁)</h4> <p>在 INNODB 存储引擎的内存结构中,对每个含有自增长值的表都有一个自增长计数器(auto-increment counter)。当对含有自增长的计数器的表进行<strong>插入操作时</strong>,这个计数器会被<strong>初始化</strong>,执行如下的语句来得到计数器的值</p> <div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="k">MAX</span> <span class="p">(</span><span class="n">auto</span> <span class="n">inc</span> <span class="n">col</span><span class="p">)</span> <span class="k">FROM</span> <span class="n">t</span> <span class="k">FOR</span> <span class="k">UPDATE</span><span class="p">:</span> </code></pre></div></div> <blockquote> <p>每张表的自増长值并不保存在磁盘上进行持久化,而是在每次 INNODB 存储引檠启动时初始化。<br /> —— 来自另一本硬核源码书</p> </blockquote> <p>插人操作会依据这个自增长的计数器值加 1 赋予自增长列。这种锁其实是采用一种特殊的表锁机制,为了提高插入的性能,<strong>锁不是在一个事务完成后オ释放,而是在完成对自增长值插入的 SQL 语句后立即释放</strong>。但对于有自增长值的列的并发插人性能较差,事务必须等待前一个插入的完成(虽然不用等待事务的完成)。</p> <h4 id="id-互斥量">互斥量</h4> <p>从 MYSQL5.1.22 版本开始,INNODB 存储引擎中提供了一种轻量级互斥量的自增长实现机制,这种机制大大提高了自增长值插入的性能。并且从该版本开始,INNODB 存储引擎提供了一个参数 <code class="language-plaintext highlighter-rouge">innodb_autoinc_lock_mode</code> 来控制自增长的模式,该参数的默认值为 1</p> <h5 id="id-三种插入类型">三种插入类型</h5> <p><img src="/img/2019-07-20-084854.png" alt="" /></p> <h5 id="id-innodb_autoinc_lock_mode-说明">innodb_autoinc_lock_mode 说明</h5> <p><img src="/img/2019-07-20-085106.png" alt="" /></p> <p>需要注意 InnoDB 存储引擎中自增长的实现和 MYISAM 不同。MYISAM 存储引擎是表锁设计,自增长不用考虑并发插人的问题。另外,在 INNODB 存储引擎中,自增长值的列必须是索引,同时必须是索引的第一个列。如果不是第一个列,则 MYSQL 数据库会抛出异常,而 MYISAM 存储引擎没有这问题。</p> <h3 id="id-外键与锁">外键与锁</h3> <p>外键主要用于引用完整性的约束检査。在 INNODB 存储引擎中,对于一个外键列,如果没有显式地对这个列加索引,INNODB 存储引擎自动对其加一个索引,因为这样可以避免表锁,Oracle 数据库不会自动添加索引,用户必须自己手动添加,这也导致了 Oracle 数据库中可能产生死锁。</p> <p>对于外键值的插入或更新,首先需要査询父表中的记录,即 SELECT 父表。但是对于父表的 SELECT 操作,不是使用一致性非锁定读的方式,因为这样会发生数据不一致的问题,因此这时使用的是 SELECT… LOCK IN SHARE MODE 方式,即主动对父表加个 S 锁。如果这时父表上已经加了 X 锁,子表上的操作会被阻塞。</p> 2019-07-20T00:00:00+00:00 https://ysmull.cn/blog/lock1.html https://ysmull.cn/blog/lock1.html Haskell 回顾(一) --- 函子、应用函子、单子 <p>距离上次学习Haskell已经过去八个多月了,以前一起讨论的同事都跑路了,<strong>谨以此系列纪念我优秀的前同事们</strong>。。。</p> <p><strong>函子:</strong>(Functor) 是把一个普通的函数映射为盒子上的函数,或者说,可以把普通函数应用到盒子内的值,列表的fmap 就是 map。</p> <div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">fmap</span> <span class="o">::</span> <span class="kt">Functor</span> <span class="n">f</span> <span class="o">=&gt;</span> <span class="p">(</span><span class="n">a</span> <span class="o">-&gt;</span> <span class="n">b</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">f</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">f</span> <span class="n">b</span> </code></pre></div></div> <p><strong>应用函子:</strong>(Applicative) 首先是个函子,额外提供如下俩操作</p> <ol> <li>把普通值包裹为盒子值,比如普通的list作为应用函子时,pure a = [a] <div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="n">pure</span> <span class="o">::</span> <span class="kt">Applicative</span> <span class="n">f</span> <span class="o">=&gt;</span> <span class="n">a</span> <span class="o">=&gt;</span> <span class="n">f</span> <span class="n">a</span> </code></pre></div> </div> </li> <li> <p>&lt;*&gt; (应用)是把盒子内的函数应用给另一个相同盒子内的值,比如 <code class="language-plaintext highlighter-rouge">[\a-&gt;a+1, \a -&gt; a * 2] &lt;*&gt; [1,2,3] 可以得到 [2,3,4,2,4,6]</code></p> <div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="p">(</span><span class="o">&lt;*&gt;</span><span class="p">)</span> <span class="o">::</span> <span class="kt">Applicative</span> <span class="n">f</span> <span class="o">=&gt;</span> <span class="n">f</span> <span class="p">(</span><span class="n">a</span> <span class="o">-&gt;</span> <span class="n">b</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">f</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="n">f</span> <span class="n">b</span> </code></pre></div> </div> </li> </ol> <p><strong>单子:</strong>(Monad) 首先是个应用函子(尽管从历史上说,是先有的Monad后有的Applicative),额外提供如下bind操作,可以把一个单子 m a 喂给另一个 (a -&gt; m b) 的函数,返回一个新的单子,比如 <code class="language-plaintext highlighter-rouge">[1,2,3] &gt;&gt;= \x -&gt; [x+1, x*2] = [2,2,3,4,4,6]</code></p> <div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="o">&gt;&gt;=</span><span class="p">)</span> <span class="o">::</span> <span class="kt">Monad</span> <span class="n">m</span> <span class="o">=&gt;</span> <span class="n">m</span> <span class="n">a</span> <span class="o">-&gt;</span> <span class="p">(</span><span class="n">a</span> <span class="o">-&gt;</span> <span class="n">m</span> <span class="n">b</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">m</span> <span class="n">b</span> </code></pre></div></div> <p>很多语言里的flatMap,在haskell中叫做 join ,可以把盒子的盒子,变成盒子类型。比如 <code class="language-plaintext highlighter-rouge">join [[1],[2],[3]] = [1,2,3]</code></p> <div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">join</span> <span class="o">::</span> <span class="kt">Monad</span> <span class="n">m</span> <span class="o">=&gt;</span> <span class="n">m</span> <span class="p">(</span><span class="n">m</span> <span class="n">a</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="n">m</span> <span class="n">a</span> <span class="n">join</span> <span class="n">x</span> <span class="o">=</span> <span class="n">x</span> <span class="o">&gt;&gt;=</span> <span class="n">id</span> <span class="p">(</span><span class="err">想想为什么这么实现</span><span class="p">)</span> </code></pre></div></div> <p><code class="language-plaintext highlighter-rouge">join</code> 和 <code class="language-plaintext highlighter-rouge">bind (&gt;&gt;=)</code> 可以互相实现:</p> <div class="language-haskell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">m</span> <span class="o">&gt;&gt;=</span> <span class="n">f</span> <span class="o">=</span> <span class="n">join</span> <span class="p">(</span><span class="n">fmap</span> <span class="n">f</span> <span class="n">m</span><span class="p">)</span> </code></pre></div></div> <p>所以在别的语言中,才会有「实现了flatMap就是Monad了」这种说法,从以上的描述看起来,这是不够严谨的,忽略了Monad同时也得是Functor,以及也要具有Applicative的操作,并且需要符合函子定律,应用函子定律和单子定律。</p> 2019-07-19T00:00:00+00:00 https://ysmull.cn/blog/hs.html https://ysmull.cn/blog/hs.html SICP - chapter 2 <ul id="markdown-toc"> <li><a href="#id-21" id="markdown-toc-id-21">2.1</a></li> <li><a href="#id-22" id="markdown-toc-id-22">2.2</a></li> <li><a href="#id-23" id="markdown-toc-id-23">2.3</a></li> <li><a href="#id-24" id="markdown-toc-id-24">2.4</a></li> <li><a href="#id-25" id="markdown-toc-id-25">2.5</a></li> <li><a href="#id-217" id="markdown-toc-id-217">2.17</a></li> <li><a href="#id-218" id="markdown-toc-id-218">2.18</a></li> <li><a href="#id-220" id="markdown-toc-id-220">2.20</a></li> <li><a href="#id-223" id="markdown-toc-id-223">2.23</a></li> <li><a href="#id-225" id="markdown-toc-id-225">2.25</a></li> <li><a href="#id-226" id="markdown-toc-id-226">2.26</a></li> <li><a href="#id-227" id="markdown-toc-id-227">2.27</a></li> <li><a href="#id-228" id="markdown-toc-id-228">2.28</a></li> </ul> <h2 id="id-21">2.1</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">make-rat</span> <span class="nv">n</span> <span class="nv">d</span><span class="p">)</span> <span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nf">t1</span> <span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="nb">&lt;</span> <span class="nv">n</span> <span class="mi">0</span><span class="p">)</span> <span class="mi">-1</span> <span class="mi">1</span><span class="p">))</span> <span class="c1">;获取n和d的符号位</span> <span class="p">(</span><span class="nf">t2</span> <span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="nb">&lt;</span> <span class="nv">d</span> <span class="mi">0</span><span class="p">)</span> <span class="mi">-1</span> <span class="mi">1</span><span class="p">)))</span> <span class="p">(</span><span class="k">cond</span> <span class="p">((</span><span class="nb">=</span> <span class="p">(</span><span class="nb">*</span> <span class="nv">t1</span> <span class="nv">t2</span><span class="p">)</span> <span class="mi">1</span><span class="p">)</span> <span class="p">(</span><span class="nb">cons</span> <span class="p">(</span><span class="nb">*</span> <span class="nv">t1</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nb">*</span> <span class="nv">t2</span> <span class="nv">d</span><span class="p">)))</span> <span class="p">((</span><span class="nb">=</span> <span class="nv">t1</span> <span class="mi">-1</span><span class="p">)</span> <span class="p">(</span><span class="nb">cons</span> <span class="nv">n</span> <span class="nv">d</span><span class="p">))</span> <span class="p">((</span><span class="nb">=</span> <span class="nv">t2</span> <span class="mi">-1</span><span class="p">)</span> <span class="p">(</span><span class="nb">cons</span> <span class="p">(</span><span class="nb">*</span> <span class="nv">t2</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nb">*</span> <span class="nv">t2</span> <span class="nv">d</span><span class="p">))))))</span> <span class="p">(</span><span class="nf">displayln</span> <span class="p">(</span><span class="nf">make-rat</span> <span class="mi">1</span> <span class="mi">22</span><span class="p">))</span> <span class="p">(</span><span class="nf">displayln</span> <span class="p">(</span><span class="nf">make-rat</span> <span class="mi">1</span> <span class="mi">-22</span><span class="p">))</span> <span class="p">(</span><span class="nf">displayln</span> <span class="p">(</span><span class="nf">make-rat</span> <span class="mi">-1</span> <span class="mi">22</span><span class="p">))</span> <span class="p">(</span><span class="nf">displayln</span> <span class="p">(</span><span class="nf">make-rat</span> <span class="mi">-1</span> <span class="mi">-22</span><span class="p">))</span> </code></pre></div></div> <h2 id="id-22">2.2</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">make-point</span> <span class="nv">x</span> <span class="nv">y</span><span class="p">)</span> <span class="p">(</span><span class="nb">cons</span> <span class="nv">x</span> <span class="nv">y</span><span class="p">))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">x-point</span> <span class="nv">p</span><span class="p">)</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">p</span><span class="p">))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">y-point</span> <span class="nv">p</span><span class="p">)</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">p</span><span class="p">))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">make-segment</span> <span class="nv">x</span> <span class="nv">y</span><span class="p">)</span> <span class="p">(</span><span class="nb">cons</span> <span class="nv">x</span> <span class="nv">y</span><span class="p">))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">start-segment</span> <span class="nv">s</span><span class="p">)</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">s</span><span class="p">))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">end-segment</span> <span class="nv">s</span><span class="p">)</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">s</span><span class="p">))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">mid-segment</span> <span class="nv">s</span><span class="p">)</span> <span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nf">startP</span> <span class="p">(</span><span class="nf">start-segment</span> <span class="nv">s</span><span class="p">))</span> <span class="p">(</span><span class="nf">endP</span> <span class="p">(</span><span class="nf">end-segment</span> <span class="nv">s</span><span class="p">)))</span> <span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nf">x1</span> <span class="p">(</span><span class="nf">x-point</span> <span class="nv">startP</span><span class="p">))</span> <span class="p">(</span><span class="nf">x2</span> <span class="p">(</span><span class="nf">x-point</span> <span class="nv">endP</span><span class="p">))</span> <span class="p">(</span><span class="nf">y1</span> <span class="p">(</span><span class="nf">y-point</span> <span class="nv">startP</span><span class="p">))</span> <span class="p">(</span><span class="nf">y2</span> <span class="p">(</span><span class="nf">y-point</span> <span class="nv">endP</span><span class="p">)))</span> <span class="p">(</span><span class="nf">make-point</span> <span class="p">(</span><span class="nb">/</span> <span class="p">(</span><span class="nb">+</span> <span class="nv">x1</span> <span class="nv">x2</span><span class="p">)</span> <span class="mi">2</span><span class="p">)</span> <span class="p">(</span><span class="nb">/</span> <span class="p">(</span><span class="nb">+</span> <span class="nv">y1</span> <span class="nv">y2</span><span class="p">)</span> <span class="mi">2</span><span class="p">)))))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">print-point</span> <span class="nv">p</span><span class="p">)</span> <span class="p">(</span><span class="nb">display</span> <span class="s">"("</span><span class="p">)</span> <span class="p">(</span><span class="nb">display</span> <span class="p">(</span><span class="nf">x-point</span> <span class="nv">p</span><span class="p">))</span> <span class="p">(</span><span class="nb">display</span> <span class="s">","</span><span class="p">)</span> <span class="p">(</span><span class="nb">display</span> <span class="p">(</span><span class="nf">y-point</span> <span class="nv">p</span><span class="p">))</span> <span class="p">(</span><span class="nb">display</span> <span class="s">")"</span><span class="p">))</span> <span class="p">(</span><span class="k">define</span> <span class="nv">seg1</span> <span class="p">(</span><span class="nf">make-segment</span> <span class="p">(</span><span class="nf">make-point</span> <span class="mi">1</span> <span class="mi">1</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-point</span> <span class="mi">3</span> <span class="mi">3</span><span class="p">)))</span> <span class="p">(</span><span class="nf">print-point</span> <span class="p">(</span><span class="nf">mid-segment</span> <span class="nv">seg1</span><span class="p">))</span> </code></pre></div></div> <h2 id="id-23">2.3</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">; 需要运行2.2</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">make-rec</span> <span class="nv">p1</span> <span class="nv">p2</span><span class="p">)</span> <span class="p">(</span><span class="nb">cons</span> <span class="nv">p1</span> <span class="nv">p2</span><span class="p">))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">rec-length</span> <span class="nv">p</span><span class="p">)</span> <span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nf">p1</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">p</span><span class="p">))</span> <span class="p">(</span><span class="nf">p2</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">p</span><span class="p">)))</span> <span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nf">x1</span> <span class="p">(</span><span class="nf">x-point</span> <span class="nv">p1</span><span class="p">))</span> <span class="p">(</span><span class="nf">x2</span> <span class="p">(</span><span class="nf">x-point</span> <span class="nv">p2</span><span class="p">)))</span> <span class="p">(</span><span class="nb">abs</span> <span class="p">(</span><span class="nb">-</span> <span class="nv">x1</span> <span class="nv">x2</span><span class="p">)))))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">rec-width</span> <span class="nv">p</span><span class="p">)</span> <span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nf">p1</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">p</span><span class="p">))</span> <span class="p">(</span><span class="nf">p2</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">p</span><span class="p">)))</span> <span class="p">(</span><span class="k">let</span> <span class="p">((</span><span class="nf">y1</span> <span class="p">(</span><span class="nf">y-point</span> <span class="nv">p1</span><span class="p">))</span> <span class="p">(</span><span class="nf">y2</span> <span class="p">(</span><span class="nf">y-point</span> <span class="nv">p2</span><span class="p">)))</span> <span class="p">(</span><span class="nb">abs</span> <span class="p">(</span><span class="nb">-</span> <span class="nv">y1</span> <span class="nv">y2</span><span class="p">)))))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">rec-area</span> <span class="nv">p</span><span class="p">)</span> <span class="p">(</span><span class="nb">*</span> <span class="p">(</span><span class="nf">rec-length</span> <span class="nv">p</span><span class="p">)</span> <span class="p">(</span><span class="nf">rec-width</span> <span class="nv">p</span><span class="p">)))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">rec-perimeter</span> <span class="nv">p</span><span class="p">)</span> <span class="p">(</span><span class="nb">*</span> <span class="mi">2</span> <span class="p">(</span><span class="nb">+</span> <span class="p">(</span><span class="nf">rec-length</span> <span class="nv">p</span><span class="p">)</span> <span class="p">(</span><span class="nf">rec-width</span> <span class="nv">p</span><span class="p">))))</span> <span class="p">(</span><span class="k">define</span> <span class="nv">rec1</span> <span class="p">(</span><span class="nf">make-rec</span> <span class="p">(</span><span class="nf">make-point</span> <span class="mi">0</span> <span class="mi">0</span><span class="p">)</span> <span class="p">(</span><span class="nf">make-point</span> <span class="mi">3</span> <span class="mi">2</span><span class="p">)))</span> <span class="p">(</span><span class="nf">displayln</span> <span class="p">(</span><span class="nf">rec-area</span> <span class="nv">rec1</span><span class="p">))</span> <span class="p">(</span><span class="nf">displayln</span> <span class="p">(</span><span class="nf">rec-perimeter</span> <span class="nv">rec1</span><span class="p">))</span> <span class="c1">; 为了使rec-area和rec-perimeter过程不改变,另一种定义方式只要提供rec-length和rec-width接口即可。</span> </code></pre></div></div> <h2 id="id-24">2.4</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">cons1</span> <span class="nv">x</span> <span class="nv">y</span><span class="p">)</span> <span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">m</span><span class="p">)</span> <span class="p">(</span><span class="nf">m</span> <span class="nv">x</span> <span class="nv">y</span><span class="p">)))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">car1</span> <span class="nv">z</span><span class="p">)</span> <span class="p">(</span><span class="nf">z</span> <span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">p</span> <span class="nv">q</span><span class="p">)</span> <span class="nv">p</span><span class="p">)))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">cdr1</span> <span class="nv">z</span><span class="p">)</span> <span class="p">(</span><span class="nf">z</span> <span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">p</span> <span class="nv">q</span><span class="p">)</span> <span class="nv">q</span><span class="p">)))</span> <span class="p">(</span><span class="nf">car1</span> <span class="p">(</span><span class="nf">cons1</span> <span class="mi">1</span> <span class="mi">2</span><span class="p">))</span> <span class="p">(</span><span class="nf">cdr1</span> <span class="p">(</span><span class="nf">cons1</span> <span class="mi">1</span> <span class="mi">2</span><span class="p">))</span> <span class="p">(</span><span class="nf">cdr1</span> <span class="p">(</span><span class="nf">cons1</span> <span class="mi">1</span> <span class="mi">2</span><span class="p">))</span> <span class="c1">; =&gt;</span> <span class="p">(</span><span class="nf">cdr1</span> <span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">m</span><span class="p">)</span> <span class="p">(</span><span class="nf">m</span> <span class="mi">1</span> <span class="mi">2</span><span class="p">)))</span> <span class="c1">; =&gt;</span> <span class="p">((</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">m</span><span class="p">)</span> <span class="p">(</span><span class="nf">m</span> <span class="mi">1</span> <span class="mi">2</span><span class="p">))</span> <span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">p</span> <span class="nv">q</span><span class="p">)</span> <span class="nv">q</span><span class="p">))</span> <span class="c1">;=&gt;</span> <span class="p">((</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">p</span> <span class="nv">q</span><span class="p">)</span> <span class="nv">q</span><span class="p">)</span> <span class="mi">1</span> <span class="mi">2</span><span class="p">)</span> </code></pre></div></div> <h2 id="id-25">2.5</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code> </code></pre></div></div> <h2 id="id-217">2.17</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="nf">displayln</span> <span class="s">"----2.17"</span><span class="p">)</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">last-pair</span> <span class="nv">l</span><span class="p">)</span> <span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="nb">null?</span> <span class="nv">l</span><span class="p">)</span> <span class="nv">null</span> <span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="nb">null?</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">l</span><span class="p">))</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">l</span><span class="p">)</span> <span class="p">(</span><span class="nf">last-pair</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">l</span><span class="p">)))))</span> <span class="p">(</span><span class="nf">last-pair</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">1</span> <span class="mi">2</span> <span class="mi">3</span><span class="p">))</span> <span class="p">(</span><span class="nf">last-pair</span> <span class="p">(</span><span class="nb">list</span><span class="p">))</span> </code></pre></div></div> <h2 id="id-218">2.18</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nb">reverse</span> <span class="nv">l</span><span class="p">)</span> <span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="nb">null?</span> <span class="nv">l</span><span class="p">)</span> <span class="nv">null</span> <span class="p">(</span><span class="nb">append</span> <span class="p">(</span><span class="nb">reverse</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">l</span><span class="p">))</span> <span class="p">(</span><span class="nb">list</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">l</span><span class="p">)))))</span> <span class="p">(</span><span class="nb">reverse</span> <span class="p">(</span><span class="nb">list</span><span class="p">))</span> <span class="p">(</span><span class="nb">reverse</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">1</span><span class="p">))</span> <span class="p">(</span><span class="nb">reverse</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">1</span> <span class="mi">2</span> <span class="mi">3</span><span class="p">))</span> </code></pre></div></div> <h2 id="id-220">2.20</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">same-parity</span> <span class="nv">x</span> <span class="o">.</span> <span class="nv">xs</span><span class="p">)</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">same-p</span> <span class="nv">n</span><span class="p">)</span> <span class="p">(</span><span class="nb">=</span> <span class="p">(</span><span class="nb">remainder</span> <span class="nv">x</span> <span class="mi">2</span><span class="p">)</span> <span class="p">(</span><span class="nb">remainder</span> <span class="nv">n</span> <span class="mi">2</span><span class="p">)))</span> <span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="nb">null?</span> <span class="nv">xs</span><span class="p">)</span> <span class="nv">null</span> <span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="nb">=</span> <span class="p">(</span><span class="nb">remainder</span> <span class="nv">x</span> <span class="mi">2</span><span class="p">)</span> <span class="p">(</span><span class="nb">remainder</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">xs</span><span class="p">)</span> <span class="mi">2</span><span class="p">))</span> <span class="p">(</span><span class="nb">cons</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">xs</span><span class="p">)</span> <span class="p">(</span><span class="nb">apply</span> <span class="nv">same-parity</span> <span class="p">(</span><span class="nb">cons</span> <span class="nv">x</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">xs</span><span class="p">))))</span> <span class="p">(</span><span class="nb">apply</span> <span class="nv">same-parity</span> <span class="p">(</span><span class="nb">cons</span> <span class="nv">x</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">xs</span><span class="p">))))))</span> <span class="p">(</span><span class="nf">same-parity</span> <span class="mi">111</span> <span class="mi">2</span> <span class="mi">3</span> <span class="mi">4</span> <span class="mi">5</span> <span class="mi">6</span> <span class="mi">7</span> <span class="mi">8</span> <span class="mi">9</span><span class="p">)</span> <span class="p">(</span><span class="nf">same-parity</span> <span class="mi">100</span> <span class="mi">2</span> <span class="mi">3</span> <span class="mi">4</span> <span class="mi">5</span> <span class="mi">6</span> <span class="mi">7</span> <span class="mi">8</span> <span class="mi">9</span><span class="p">)</span> </code></pre></div></div> <h2 id="id-223">2.23</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nb">for-each</span> <span class="nv">f</span> <span class="nv">xs</span><span class="p">)</span> <span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="nb">null?</span> <span class="nv">xs</span><span class="p">)</span> <span class="no">#t</span> <span class="p">(</span><span class="k">begin</span> <span class="p">(</span><span class="nf">f</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">xs</span><span class="p">))</span> <span class="p">(</span><span class="nb">for-each</span> <span class="nv">f</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">xs</span><span class="p">)))))</span> <span class="p">(</span><span class="nb">for-each</span> <span class="p">(</span><span class="k">lambda</span> <span class="p">(</span><span class="nf">x</span><span class="p">)</span> <span class="p">(</span><span class="nf">displayln</span> <span class="nv">x</span><span class="p">))</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">1</span> <span class="mi">2</span> <span class="mi">3</span><span class="p">))</span> </code></pre></div></div> <h2 id="id-225">2.25</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">define</span> <span class="nv">x25-1</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">1</span> <span class="mi">3</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">5</span> <span class="mi">7</span><span class="p">)</span> <span class="mi">9</span><span class="p">))</span> <span class="p">(</span><span class="nb">car</span> <span class="p">(</span><span class="nb">cdr</span> <span class="p">(</span><span class="nb">car</span> <span class="p">(</span><span class="nb">cdr</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">x25-1</span><span class="p">)))))</span> <span class="p">(</span><span class="k">define</span> <span class="nv">x25-2</span> <span class="p">(</span><span class="nb">list</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">7</span><span class="p">)))</span> <span class="p">(</span><span class="nb">car</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">x25-2</span><span class="p">))</span> <span class="p">(</span><span class="k">define</span> <span class="nv">x25-3</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">1</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">2</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">3</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">4</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">5</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">6</span> <span class="mi">7</span><span class="p">)))))))</span> <span class="p">(</span><span class="nb">car</span> <span class="p">(</span><span class="nb">cdr</span> <span class="p">(</span><span class="nb">car</span> <span class="p">(</span><span class="nb">cdr</span> <span class="p">(</span><span class="nb">car</span> <span class="p">(</span><span class="nb">cdr</span> <span class="p">(</span><span class="nb">car</span> <span class="p">(</span><span class="nb">cdr</span> <span class="p">(</span><span class="nb">car</span> <span class="p">(</span><span class="nb">cdr</span> <span class="p">(</span><span class="nb">car</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">x25-3</span><span class="p">))))))))))))</span> </code></pre></div></div> <h2 id="id-226">2.26</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">define</span> <span class="nv">x26</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">1</span> <span class="mi">2</span> <span class="mi">3</span><span class="p">))</span> <span class="p">(</span><span class="k">define</span> <span class="nv">y26</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">4</span> <span class="mi">5</span> <span class="mi">6</span><span class="p">))</span> <span class="p">(</span><span class="nb">append</span> <span class="nv">x26</span> <span class="nv">y26</span><span class="p">)</span> <span class="p">(</span><span class="nb">cons</span> <span class="nv">x26</span> <span class="nv">y26</span><span class="p">)</span> <span class="p">(</span><span class="nb">list</span> <span class="nv">x26</span> <span class="nv">y26</span><span class="p">)</span> </code></pre></div></div> <h2 id="id-227">2.27</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">deep-reverse</span> <span class="nv">l</span><span class="p">)</span> <span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="nb">null?</span> <span class="nv">l</span><span class="p">)</span> <span class="nv">null</span> <span class="p">(</span><span class="k">if</span> <span class="p">(</span><span class="nb">list?</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">l</span><span class="p">))</span> <span class="p">(</span><span class="nb">append</span> <span class="p">(</span><span class="nf">deep-reverse</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">l</span><span class="p">))</span> <span class="p">(</span><span class="nb">list</span> <span class="p">(</span><span class="nf">deep-reverse</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">l</span><span class="p">))))</span> <span class="p">(</span><span class="nb">append</span> <span class="p">(</span><span class="nf">deep-reverse</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">l</span><span class="p">))</span> <span class="p">(</span><span class="nb">list</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">l</span><span class="p">))))))</span> <span class="p">(</span><span class="nf">deep-reverse</span> <span class="o">'</span><span class="p">())</span> <span class="p">(</span><span class="nf">deep-reverse</span> <span class="o">'</span><span class="p">(</span><span class="nf">1</span><span class="p">))</span> <span class="p">(</span><span class="nf">deep-reverse</span> <span class="o">'</span><span class="p">(</span><span class="nf">1</span> <span class="mi">2</span><span class="p">))</span> <span class="p">(</span><span class="nf">deep-reverse</span> <span class="p">(</span><span class="nb">list</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">1</span> <span class="mi">2</span><span class="p">)</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">3</span> <span class="mi">4</span><span class="p">)))</span> <span class="p">(</span><span class="nf">deep-reverse</span> <span class="o">'</span><span class="p">((</span><span class="nf">1</span> <span class="mi">2</span><span class="p">)</span> <span class="p">(</span><span class="nf">3</span> <span class="mi">4</span><span class="p">)</span> <span class="mi">5</span> <span class="mi">6</span> <span class="p">(</span><span class="nf">7</span> <span class="mi">8</span><span class="p">)))</span> </code></pre></div></div> <h2 id="id-228">2.28</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="nf">displayln</span> <span class="s">"二叉树版抽象"</span><span class="p">)</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">left-tree</span> <span class="nv">tree</span><span class="p">)</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">tree</span><span class="p">))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">right-tree</span> <span class="nv">tree</span><span class="p">)</span> <span class="p">(</span><span class="nb">car</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">tree</span><span class="p">)))</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">fringe-bin</span> <span class="nv">tree</span><span class="p">)</span> <span class="p">(</span><span class="k">cond</span> <span class="p">((</span><span class="nb">null?</span> <span class="nv">tree</span><span class="p">)</span> <span class="nv">null</span><span class="p">)</span> <span class="p">((</span><span class="nb">pair?</span> <span class="nv">tree</span><span class="p">)</span> <span class="p">(</span><span class="nb">append</span> <span class="p">(</span><span class="nf">fringe-bin</span> <span class="p">(</span><span class="nf">left-tree</span> <span class="nv">tree</span><span class="p">))</span> <span class="p">(</span><span class="nf">fringe-bin</span> <span class="p">(</span><span class="nf">right-tree</span> <span class="nv">tree</span><span class="p">))))</span> <span class="p">(</span><span class="k">else</span> <span class="p">(</span><span class="nb">list</span> <span class="nv">tree</span><span class="p">))))</span> <span class="c1">; 多叉树无法work</span> <span class="c1">;(fringe-bin (list (list 1 2 3) (list 4 5 6) (list 7 8) 9))</span> <span class="p">(</span><span class="k">define</span> <span class="nv">x28</span> <span class="p">(</span><span class="nb">list</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">1</span> <span class="mi">2</span><span class="p">)</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">3</span> <span class="mi">4</span><span class="p">)))</span> <span class="p">(</span><span class="nf">fringe-bin</span> <span class="nv">x28</span><span class="p">)</span> <span class="p">(</span><span class="nf">fringe-bin</span> <span class="p">(</span><span class="nb">list</span> <span class="nv">x28</span> <span class="nv">x28</span><span class="p">))</span> <span class="c1">; 多叉树work版</span> <span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">fringe</span> <span class="nv">tree</span><span class="p">)</span> <span class="p">(</span><span class="k">cond</span> <span class="p">((</span><span class="nb">null?</span> <span class="nv">tree</span><span class="p">)</span> <span class="nv">null</span><span class="p">)</span> <span class="p">((</span><span class="nb">pair?</span> <span class="nv">tree</span><span class="p">)</span> <span class="p">(</span><span class="nb">append</span> <span class="p">(</span><span class="nf">fringe</span> <span class="p">(</span><span class="nb">car</span> <span class="nv">tree</span><span class="p">))</span> <span class="p">(</span><span class="nf">fringe</span> <span class="p">(</span><span class="nb">cdr</span> <span class="nv">tree</span><span class="p">))))</span> <span class="p">(</span><span class="k">else</span> <span class="p">(</span><span class="nb">list</span> <span class="nv">tree</span><span class="p">))))</span> <span class="p">(</span><span class="nf">fringe</span> <span class="p">(</span><span class="nb">list</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">1</span> <span class="mi">2</span> <span class="mi">3</span><span class="p">)</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">4</span> <span class="mi">5</span> <span class="mi">6</span><span class="p">)</span> <span class="p">(</span><span class="nb">list</span> <span class="mi">7</span> <span class="mi">8</span><span class="p">)</span> <span class="mi">9</span><span class="p">))</span> </code></pre></div></div> 2018-05-04T00:00:00+00:00 https://ysmull.cn/blog/sicp2.html https://ysmull.cn/blog/sicp2.html SICP - chapter 1 <ul id="markdown-toc"> <li><a href="#id-13" id="markdown-toc-id-13">1.3</a></li> <li><a href="#id-14" id="markdown-toc-id-14">1.4</a></li> <li><a href="#id-15" id="markdown-toc-id-15">1.5</a></li> <li><a href="#id-16" id="markdown-toc-id-16">1.6</a></li> <li><a href="#id-17" id="markdown-toc-id-17">1.7</a></li> <li><a href="#id-18" id="markdown-toc-id-18">1.8</a></li> <li><a href="#id-112" id="markdown-toc-id-112">1.12</a></li> <li><a href="#id-121" id="markdown-toc-id-121">1.21</a></li> <li><a href="#id-122" id="markdown-toc-id-122">1.22</a></li> <li><a href="#id-123" id="markdown-toc-id-123">1.23</a></li> <li><a href="#id-124" id="markdown-toc-id-124">1.24</a></li> <li><a href="#id-127" id="markdown-toc-id-127">1.27</a></li> <li><a href="#id-128" id="markdown-toc-id-128">1.28</a></li> </ul> <h2 id="id-13">1.3</h2> <div class="language-scheme highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="k">define</span> <span class="p">(</span><span class="nf">max_tow_sum</span> <span class="nv">a</span> <span class="nv">b</span> <span class="nv">c</span><span class="p">)</span> <span class="p">(</span><span class="k">cond</span> <span class="p">((</span><span class="k">and</span> <span class="p">(</span><span class="nb">&gt;=</span> <span class="nv">a</span> <span class="nv">b</span><span class="p">)</span> <span class="p">(</span><span class="nb">&gt;=</span> <span class="nv">b</span> <span class="nv">c</span><span class="p">))</span> <span class="p">(</span><span class="nb">+</span> <span class="nv">a</span> <span class="nv">b</span><span class="p">))</span> <span class="p">((</span><span class="k">and</span> <span class="p">(</span><span class="nb">&gt;=</span> <span class="nv">a</span> <span class="nv">c</span><span class="p">)</span> <span class="p">(</span><span class="nb">&gt;=</span> <span class="nv">c</span> <span class="nv">b</span><span class="p">))</span> <span class="p">(</span><span class="nb">+</span> <span class="nv">a</span> <span class="nv">c</span><span class="p">))</span> <span class="p">((</span><span class="k">and</span> <span class="p">(</span><span class="nb">&gt;=</span> <span class="nv">b</span> <span class="nv">a</span><span class="p">)</span> <span class="p">(</span><span class="nb">&gt;=<