xinyin025/hello-algo
1
0
Fork 0
mirror of https://github.com/krahets/hello-algo.git synced 2026-07-13 23:56:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

deploy

Browse Source
This commit is contained in:
krahets
2026-04-02 03:08:50 +08:00
parent 09a136c9fa
commit aaf9f58eb3
157 changed files with 3002 additions and 2994 deletions
Download Patch File Download Diff File Expand all files Collapse all files
en/chapter_data_structure/character_encoding/index.html
+25 -25
View File
@@ -6,7 +6,7 @@
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<meta name="description" content="Data Structures and Algorithms Crash Course with Animated Illustrations and Off-the-Shelf Code">
<meta name="description" content="Data structures and algorithms tutorial with animated illustrations and ready-to-run code">
<meta name="author" content="krahets">
@@ -576,7 +576,7 @@
<span class="md-ellipsis">
Chapter 1. Encounter With Algorithms
Chapter 1. Encounter with Algorithms
@@ -598,7 +598,7 @@
<span class="md-nav__icon md-icon"></span>
Chapter 1. Encounter With Algorithms
Chapter 1. Encounter with Algorithms
</label>
@@ -1288,7 +1288,7 @@
<span class="md-ellipsis">
Chapter 4. Array and Linked List
Chapter 4. Arrays and Linked Lists
@@ -1310,7 +1310,7 @@
<span class="md-nav__icon md-icon"></span>
Chapter 4. Array and Linked List
Chapter 4. Arrays and Linked Lists
</label>
@@ -1416,7 +1416,7 @@
<span class="md-ellipsis">
4.4 Memory and Cache *
4.4 Random-Access Memory and Cache *
@@ -1507,7 +1507,7 @@
<span class="md-ellipsis">
Chapter 5. Stack and Queue
Chapter 5. Stacks and Queues
@@ -1529,7 +1529,7 @@
<span class="md-nav__icon md-icon"></span>
Chapter 5. Stack and Queue
Chapter 5. Stacks and Queues
</label>
@@ -1607,7 +1607,7 @@
<span class="md-ellipsis">
5.3 Double-Ended Queue
5.3 Deque
@@ -1698,7 +1698,7 @@
<span class="md-ellipsis">
Chapter 6. Hashing
Chapter 6. Hash Table
@@ -1720,7 +1720,7 @@
<span class="md-nav__icon md-icon"></span>
Chapter 6. Hashing
Chapter 6. Hash Table
</label>
@@ -1993,7 +1993,7 @@
<span class="md-ellipsis">
7.3 Array Representation of Tree
7.3 Array Representation of Binary Trees
@@ -2212,7 +2212,7 @@
<span class="md-ellipsis">
8.2 Building a Heap
8.2 Heap Construction Operation
@@ -2240,7 +2240,7 @@
<span class="md-ellipsis">
8.3 Top-K Problem
8.3 Top-k Problem
@@ -2598,7 +2598,7 @@
<span class="md-ellipsis">
10.2 Binary Search Insertion
10.2 Binary Search Insertion Point
@@ -2626,7 +2626,7 @@
<span class="md-ellipsis">
10.3 Binary Search Edge Cases
10.3 Binary Search Boundaries
@@ -2682,7 +2682,7 @@
<span class="md-ellipsis">
10.5 Search Algorithms Revisited
10.5 Searching Algorithms Revisited
@@ -2831,7 +2831,7 @@
<span class="md-ellipsis">
11.1 Sorting Algorithms
11.1 Sorting Algorithm
@@ -3304,7 +3304,7 @@
<span class="md-ellipsis">
12.4 Hanoi Tower Problem
12.4 Hanota Problem
@@ -4216,7 +4216,7 @@
<span class="md-ellipsis">
16.3 Terminology Table
16.3 Glossary
@@ -4432,13 +4432,13 @@
<p>However, <strong>ASCII code can only represent English</strong>. With the globalization of computers, a character set called <u>EASCII</u> that can represent more languages emerged. It expands from the 7-bit basis of ASCII to 8 bits, and can represent 256 different characters.</p>
<p>Worldwide, a batch of EASCII character sets suitable for different regions have appeared successively. The first 128 characters of these character sets are unified as ASCII code, and the last 128 characters are defined differently to adapt to the needs of different languages.</p>
<h2 id="342-gbk-character-set">3.4.2 &nbsp; Gbk Character Set<a class="headerlink" href="#342-gbk-character-set" title="Permanent link">&para;</a></h2>
<p>Later, people found that <strong>EASCII code still cannot meet the character quantity requirements of many languages</strong>. For example, there are nearly one hundred thousand Chinese characters, and several thousand are used daily. In 1980, the China National Standardization Administration released the <u>GB2312</u> character set, which included 6,763 Chinese characters, basically meeting the needs for computer processing of Chinese characters.</p>
<p>Later, people found that <strong>EASCII still could not provide enough characters for many languages</strong>. For example, there are nearly one hundred thousand Chinese characters, and several thousand are used in everyday life. In 1980, the China National Standardization Administration released the <u>GB2312</u> character set, which included 6,763 Chinese characters, basically meeting the needs of computer processing for Chinese.</p>
<p>However, GB2312 cannot handle some rare characters and traditional Chinese characters. The <u>GBK</u> character set is an extension based on GB2312, which includes a total of 21,886 Chinese characters. In the GBK encoding scheme, ASCII characters are represented using one byte, and Chinese characters are represented using two bytes.</p>
<h2 id="343-unicode-character-set">3.4.3 &nbsp; Unicode Character Set<a class="headerlink" href="#343-unicode-character-set" title="Permanent link">&para;</a></h2>
<p>With the vigorous development of computer technology, character sets and encoding standards flourished, which brought many problems. On the one hand, these character sets generally only define characters for specific languages and cannot work normally in multilingual environments. On the other hand, multiple character set standards exist for the same language, and if two computers use different encoding standards, garbled characters will appear during information transmission.</p>
<p>Researchers of that era thought: <strong>If a sufficiently complete character set is released that includes all languages and symbols in the world, wouldn't it be possible to solve cross-language environment and garbled character problems</strong>? Driven by this idea, a large and comprehensive character set, Unicode, was born.</p>
<p>Researchers of that era thought: <strong>If a sufficiently complete character set were released to include all languages and symbols in the world, wouldn't that solve problems in cross-language environments and eliminate garbled text</strong>? Driven by this idea, a large and comprehensive character set, Unicode, was born.</p>
<p><u>Unicode</u> is called "统一码" (Unified Code) in Chinese and can theoretically accommodate over one million characters. It is committed to including characters from around the world into a unified character set, providing a universal character set to handle and display various language texts, reducing garbled character problems caused by different encoding standards.</p>
<p>Since its release in 1991, Unicode has continuously expanded to include new languages and characters. As of September 2022, Unicode has included 149,186 characters, including characters, symbols, and even emojis from various languages. In the vast Unicode character set, commonly used characters occupy 2 bytes, and some rare characters occupy 3 bytes or even 4 bytes.</p>
<p>Since its release in 1991, Unicode has continuously expanded to include new languages and characters. As of September 2022, Unicode has included 149,186 characters, including characters, symbols, and even emojis from various languages. In practical storage and encoding schemes for this vast character set, commonly used characters often occupy 2 bytes, while some rare characters occupy 3 bytes or even 4 bytes.</p>
<p>Unicode is a universal character set that essentially assigns a number (called a "code point") to each character, <strong>but it does not specify how to store these character code points in computers</strong>. We can't help but ask: when Unicode code points of multiple lengths appear simultaneously in a text, how does the system parse the characters? For example, given an encoding with a length of 2 bytes, how does the system determine whether it is one 2-byte character or two 1-byte characters?</p>
<p>For the above problem, <strong>a straightforward solution is to store all characters as equal-length encodings</strong>. As shown in Figure 3-7, each character in "Hello" occupies 1 byte, and each character in "算法" (algorithm) occupies 2 bytes. We can encode all characters in "Hello 算法" as 2 bytes in length by padding the high bits with 0. In this way, the system can parse one character every 2 bytes and restore the content of this phrase.</p>
<p><img alt="Unicode encoding example" class="animation-figure" src="../character_encoding.assets/unicode_hello_algo.png" /></p>
@@ -4452,7 +4452,7 @@
<li>For 1-byte characters, set the highest bit to <span class="arithmatex">\(0\)</span>, and set the remaining 7 bits to the Unicode code point. It is worth noting that ASCII characters occupy the first 128 code points in the Unicode character set. That is to say, <strong>UTF-8 encoding is backward compatible with ASCII code</strong>. This means we can use UTF-8 to parse very old ASCII code text.</li>
<li>For characters with a length of <span class="arithmatex">\(n\)</span> bytes (where <span class="arithmatex">\(n &gt; 1\)</span>), set the highest <span class="arithmatex">\(n\)</span> bits of the first byte to <span class="arithmatex">\(1\)</span>, and set the <span class="arithmatex">\((n + 1)\)</span>-th bit to <span class="arithmatex">\(0\)</span>; starting from the second byte, set the highest 2 bits of each byte to <span class="arithmatex">\(10\)</span>; use all remaining bits to fill in the Unicode code point of the character.</li>
</ul>
<p>Figure 3-8 shows the UTF-8 encoding corresponding to "Hello算法". It can be observed that since the highest <span class="arithmatex">\(n\)</span> bits are all set to <span class="arithmatex">\(1\)</span>, the system can parse the length of the character as <span class="arithmatex">\(n\)</span> by reading the number of highest bits that are <span class="arithmatex">\(1\)</span>.</p>
<p>Figure 3-8 shows the UTF-8 encoding corresponding to "Hello算法". It can be observed that since the highest <span class="arithmatex">\(n\)</span> bits are all set to <span class="arithmatex">\(1\)</span>, the system can determine that the character length is <span class="arithmatex">\(n\)</span> by counting the leading <span class="arithmatex">\(1\)</span> bits.</p>
<p>But why set the highest 2 bits of all other bytes to <span class="arithmatex">\(10\)</span>? In fact, this <span class="arithmatex">\(10\)</span> can serve as a check symbol. Assuming the system starts parsing text from an incorrect byte, the <span class="arithmatex">\(10\)</span> at the beginning of the byte can help the system quickly determine an anomaly.</p>
<p>The reason for using <span class="arithmatex">\(10\)</span> as a check symbol is that under UTF-8 encoding rules, it is impossible for a character's highest two bits to be <span class="arithmatex">\(10\)</span>. This conclusion can be proven by contradiction: assuming the highest two bits of a character are <span class="arithmatex">\(10\)</span>, it means the length of the character is <span class="arithmatex">\(1\)</span>, corresponding to ASCII code. However, the highest bit of ASCII code should be <span class="arithmatex">\(0\)</span>, which contradicts the assumption.</p>
<p><img alt="UTF-8 encoding example" class="animation-figure" src="../character_encoding.assets/utf-8_hello_algo.png" /></p>
@@ -4466,7 +4466,7 @@
<p>From the perspective of storage space occupation, using UTF-8 to represent English characters is very efficient because it only requires 1 byte; using UTF-16 encoding for some non-English characters (such as Chinese) will be more efficient because it only requires 2 bytes, while UTF-8 may require 3 bytes.</p>
<p>From a compatibility perspective, UTF-8 has the best universality, and many tools and libraries support UTF-8 first.</p>
<h2 id="345-character-encoding-in-programming-languages">3.4.5 &nbsp; Character Encoding in Programming Languages<a class="headerlink" href="#345-character-encoding-in-programming-languages" title="Permanent link">&para;</a></h2>
<p>For most past programming languages, strings during program execution use fixed-length encodings such as UTF-16 or UTF-32. Under fixed-length encoding, we can treat strings as arrays for processing, and this approach has the following advantages.</p>
<p>For many programming languages in the past, strings during program execution used internal encodings such as UTF-16 or UTF-32. Under these representations, we can often treat strings like arrays during processing, and this approach has the following advantages.</p>
<ul>
<li><strong>Random access</strong>: UTF-16 encoded strings can be easily accessed randomly. UTF-8 is a variable-length encoding. To find the <span class="arithmatex">\(i\)</span>-th character, we need to traverse from the beginning of the string to the <span class="arithmatex">\(i\)</span>-th character, which requires <span class="arithmatex">\(O(n)\)</span> time.</li>
<li><strong>Character counting</strong>: Similar to random access, calculating the length of a UTF-16 encoded string is also an <span class="arithmatex">\(O(1)\)</span> operation. However, calculating the length of a UTF-8 encoded string requires traversing the entire string.</li>