News·SN67 Harnyx Medium·Jan 21, 2026, 08:00 UTC
Fundamentals of the Transformer Architecture Input Embedding & Positional Encoding Layers
About SN67 Harnyx
In this article, I will attempt to address the Input Embedding and Positional Encoding sections of the Transformers architecture. Let’s try to understand the architecture step by step. Before moving on to the concept of Input Embedding, let’s try to understand what Input is. Simply put, Input is the input text we give to the model. Of course, it is possible to train with different data types, but we will give our examples based on text data. Above, we see an example sentence that will be input into our model. Our model cannot understand text as it is; we need to convert this text into numerical representations and feed our model in this way. Therefore, we first tokenize the sentences, meaning we break them down into words or smaller pieces (often tokenization is performed in different ways rather than word by word). The example text above has been tokenized word by word. After tokenization, we need to convert these words into numerical representations. We map words to numbers, and these numbers represent their positions in our vocabulary. Imagine we have a vocabulary consisting of all possible words that appear in our training set; each word will have a position in this vocabulary. For example, the word “cat” might occupy the 3578th position. After the tokenization process described above, each word is represented by a number based on its position in the vocabulary. These numbers, or InputIDs, are mapped to a 512-dimensional vector. (This vector size varies depending on the model architecture.) These 512-dimensional vectors are called embeddings. The same word is always mapped to the same InputID and embedding. However, the embedding numbers are not fixed; they are parameters for our model. Our model learns to change these numbers to represent meaning. As a result, the InputID values remain constant because the vocabulary is fixed, but the embeddings will change during the training process according to the needs of the loss function. Note: Random numbers have been a
Read the article at SN67 Harnyx Medium ↗
SN67 Harnyx at a glance
SN67, Harnyx (α token ط), is one of the independent markets on the Bittensor network, each with its own α token and its own price. Its α currently trades at 0.004360 τ (≈ $0.99), against 2,951 τ of pool liquidity. On-chain, the best-delegated validator on it yields about 119.5% APR, emissions add 0.000359 τ to the pool each block, epochs run every 360 blocks. It was registered at block #7,236,936.
More from SN67 Harnyx
Taosis indexes announcements, articles and releases about Bittensor subnets from their own channels and keeps a record of each. The text above is the source’s; the figures are read from the chain by Taosis.