A Novel Approach for Text Encryption Using Tokenizers in Ruby

# frozen_string_literal: true



require 'tokenizers'

require 'openssl'

require 'base64'

require 'securerandom'

require 'json'



# SecureTextCipher implements an enhanced reversible text encryption

# algorithm based on tokenization and secure token mapping.

class SecureTextCipher

  AES_KEY_SIZE = 32 # 256-bit key

  AES_MODE = 'AES-256-CBC'



  # Initializes the cipher with a tokenizer and a secret key.

  #

  # @param tokenizer [Object] a pre-trained tokenizer (default: BERT uncased)

  # @param secret_key [String] a secret key for HMAC and AES encryption (random by default)

  def initialize(tokenizer: nil, secret_key: nil)

    @tokenizer = tokenizer || Tokenizers::Tokenizer.from_pretrained("bert-base-uncased")

    @secret_key = secret_key || SecureRandom.random_bytes(AES_KEY_SIZE)

  end



  # Encrypts the given text by tokenizing it and remapping token IDs to secure, random IDs.

  #

  # The process includes:

  #   1. Tokenizing the text.

  #   2. For each unique token ID, generating a new secure ID using an HMAC (with a salt and a counter).

  #   3. Creating an encrypted string from the new token IDs.

  #   4. Generating a decode matrix (encrypted using AES) for reversing the mapping.

  #

  # @param text [String] the text to encode.

  # @return [Array(String, String, String)] the encrypted token string, the encrypted decode matrix, and the salt.

  def encode(text)

    encoding = @tokenizer.encode(text)

    original_ids = encoding.ids



    salt = SecureRandom.hex(8) # Salt for randomizing the mapping

    mapping = {}

    used_new_ids = {}



    original_ids.each do |token_id|

      # Skip if a mapping already exists for this token.

      next if mapping.key?(token_id)



      new_id = nil

      counter = 0

      loop do

        candidate = generate_hmac_id(token_id, salt, counter)

        unless used_new_ids.key?(candidate)

          new_id = candidate

          break

        end

        counter += 1

      end



      mapping[token_id] = new_id

      used_new_ids[new_id] = true

    end



    encrypted_ids = original_ids.map { |token_id| mapping[token_id] }

    encrypted_string = encrypted_ids.join(' ')

    decode_matrix = mapping.invert

    encrypted_matrix = encrypt_decode_matrix(decode_matrix)



    return encrypted_string, encrypted_matrix, salt

  end



  # Decrypts the encrypted token string back into the original text.

  #

  # The process includes:

  #   1. Decrypting the stored decode matrix.

  #   2. Splitting the encrypted string into token IDs.

  #   3. Mapping these secure IDs back to the original token IDs.

  #   4. Decoding the token IDs back into text.

  #

  # @param encrypted_string [String] the encrypted token IDs (space-separated).

  # @param encrypted_matrix [String] the AES-encrypted decode matrix.

  # @param salt [String] the salt used during encoding.

  # @return [String] the decoded original text.

  def decode(encrypted_string, encrypted_matrix, salt)

    # The salt is maintained to ensure consistency though not directly used here.

    decode_matrix = decrypt_decode_matrix(encrypted_matrix)

    new_ids = encrypted_string.split(' ').map(&:to_i)

    original_ids = new_ids.map { |new_id| decode_matrix[new_id] }

    @tokenizer.decode(original_ids, skip_special_tokens: true)

  end



  private



  # Generates a secure token ID using HMAC with a salt and a counter to avoid collisions.

  #

  # @param token_id [Integer] the original token ID.

  # @param salt [String] the salt value.

  # @param counter [Integer] a counter to ensure uniqueness.

  # @return [Integer] a new secure token ID.

  def generate_hmac_id(token_id, salt, counter)

    data = "#{token_id}:#{salt}:#{counter}"

    hmac = OpenSSL::HMAC.digest('SHA256', @secret_key, data)

    # Unpack to a 32-bit unsigned integer and limit the range

    hmac.unpack1('L*') % 10_000

  end



  # Encrypts the decode matrix (a hash) using AES-256-CBC and returns a Base64 encoded string.

  #

  # @param matrix [Hash] the decode matrix mapping new IDs to original token IDs.

  # @return [String] the encrypted and Base64 encoded decode matrix.

  def encrypt_decode_matrix(matrix)

    cipher = OpenSSL::Cipher.new(AES_MODE)

    cipher.encrypt

    iv = cipher.random_iv

    cipher.key = @secret_key



    json_matrix = matrix.to_json

    encrypted = cipher.update(json_matrix) + cipher.final



    Base64.strict_encode64(iv + encrypted)

  end



  # Decrypts the AES-encrypted decode matrix.

  #

  # @param encrypted_matrix [String] the Base64 encoded encrypted matrix.

  # @return [Hash] the decrypted decode matrix with integer keys.

  def decrypt_decode_matrix(encrypted_matrix)

    decoded = Base64.strict_decode64(encrypted_matrix)

    iv = decoded[0..15] # First 16 bytes are the IV

    encrypted_data = decoded[16..-1]



    cipher = OpenSSL::Cipher.new(AES_MODE)

    cipher.decrypt

    cipher.key = @secret_key

    cipher.iv = iv



    json_matrix = cipher.update(encrypted_data) + cipher.final

    # Convert keys back to integers after JSON parsing.

    JSON.parse(json_matrix).transform_keys { |k| k.to_i }

  end

end



# --- Example Usage ---

if __FILE__ == $PROGRAM_NAME

  cipher = SecureTextCipher.new

  original_text = "Hello, world! Secure encryption in Ruby."



  encrypted_string, encrypted_matrix, salt = cipher.encode(original_text)

  decrypted_text = cipher.decode(encrypted_string, encrypted_matrix, salt)



  puts "Original Text: #{original_text}"

  puts "Encrypted String: #{encrypted_string}"

  puts "Decrypted Text: #{decrypted_text}"

end

Abstract

In this paper, I present a novel approach for reversible text encryption using tokenization techniques in Ruby. My method leverages the tokenizers gem to tokenize input text and then securely remaps each token’s identifier to a new, randomly generated identifier. To ensure reversibility, I maintain an encrypted decoding matrix using AES-256-CBC. Additional security is achieved by generating secure IDs using HMAC with a salt and counter mechanism. This approach provides a lightweight yet enhanced security solution for scenarios where traditional cryptographic methods may be excessive.

1. Introduction

1.1 Motivation

Text encryption is critical in many modern applications. Although traditional encryption algorithms such as AES and RSA offer robust security, I found them sometimes unnecessarily complex for tasks that only require reversible obfuscation of text. My approach focuses on a token-based method that efficiently obfuscates text by leveraging tokenization, HMAC-based remapping, and symmetric encryption for secure storage of mapping information.

1.2 Contributions

The key contributions of my work are:

• 
  Token-based Encryption: I encrypt tokenized text rather than raw text.


• 
  Secure Token Mapping: I use HMAC with a salt and counter to generate secure, non-colliding token IDs.


• 
  Encrypted Decode Matrix: I utilize AES-256-CBC to protect the mapping required for decryption.


• 
  Lightweight and Reversible: I offer a reversible transformation with minimal computational overhead.

2. Background

2.1 Tokenization in Natural Language Processing

Tokenization converts text into a sequence of tokens, which can represent words, subwords, or characters. Models such as BERT rely on tokenization to convert text into numerical formats for further processing. I leverage the tokenizers gem in Ruby, which provides an efficient interface to these pre-trained tokenizers.

2.2 Cryptographic Primitives Employed

• 
  HMAC: I use HMAC to generate secure, unique token IDs by hashing token information with a secret key and salt.


• 
  AES-256-CBC: I employ this symmetric encryption algorithm to securely encrypt the decode matrix, ensuring that the mapping information is protected from unauthorized access.


• 
  Salt and Counter: These elements ensure that even identical token IDs yield unique secure IDs on different encryptions, mitigating collision risks.

3. Methodology

3.1 Overview of the Encryption Process

The encryption process follows these steps:

1. 
   Tokenization: I convert the input text into token IDs using a pre-trained tokenizer.


2. 
   Secure Mapping Generation: For each unique token ID, I generate a new secure token ID using HMAC combined with a salt and a counter to avoid collisions.


3. 
   Token Substitution: I replace the original token IDs with the newly generated secure IDs.


4. 
   Decoding Matrix Construction: I create a reverse mapping (decode matrix) and encrypt it using AES-256-CBC.


5. 
   Output: I produce an encrypted string (the secure token IDs) and an encrypted decode matrix, along with the salt used for HMAC generation.

3.2 Decryption Process

The decryption process reverses the encryption steps:

1. 
   Decrypt the Decode Matrix: I use the stored AES key to decrypt the decode matrix.


2. 
   Token ID Restoration: I convert the secure token IDs back to the original token IDs using the decode matrix.


3. 
   Text Reconstruction: I decode the original token IDs back into text using the tokenizer’s decode function.

3.3 Handling Collisions

To ensure that each token is mapped to a unique secure ID, I use a counter appended to the HMAC input. If a collision is detected (i.e., a secure ID already exists), I increment the counter and generate a new secure ID until I obtain a unique one.

4. Implementation Details

I provide a complete implementation in Ruby. The key components include:

• 
  HMAC-based ID Generation: I securely compute new token IDs while handling potential collisions.


• 
  AES-256-CBC Encryption of the Decode Matrix: I ensure that the reverse mapping is stored securely.


• 
  Tokenizers Gem Integration: I utilize a pre-trained BERT tokenizer to perform efficient tokenization and decoding.

5. Experimental Results

5.1 Test Cases

I tested the algorithm with:

• Short sentences containing punctuation.


• Longer passages with complex structures.


• Edge cases such as numbers and special characters.

5.2 Performance

The encryption and decryption processes execute in near real-time for typical text sizes. My method is computationally lightweight, making it suitable for real-time applications.

5.3 Reversibility and Security

All test cases confirmed that the decryption process accurately recovered the original text. The use of HMAC with salt and a counter, along with AES encryption for the decode matrix, significantly enhances security compared to a simple random mapping approach.

6. Security Analysis

6.1 Strengths

• 
  Reversible Transformation: The original text is perfectly recoverable using the encrypted decode matrix.


• 
  Secure Token Mapping: HMAC-based token ID generation, combined with salt and a counter, minimizes collision risks and makes it difficult for an attacker to predict the mapping.


• 
  Encryption of the Decode Matrix: AES-256-CBC encryption protects the reverse mapping from unauthorized access.

6.2 Limitations

• 
  Not a Replacement for Full Cryptographic Schemes: Although my method adds several layers of security, it is designed for lightweight reversible obfuscation rather than high-security applications.


• 
  Key Management: The security of the system depends on my ability to safeguard the secret key used for both HMAC and AES encryption.

7. Future Work

Future improvements I plan to explore include:

• 
  Integrating Additional Cryptographic Measures: I will consider combining token-based encryption with traditional methods for enhanced security.


• 
  Multilingual Support: I aim to adapt the approach to work with different tokenizers and languages.


• 
  Dynamic Key Rotation: I plan to implement secure key management and periodic rotation to further enhance security.

8. Conclusion

I have introduced a secure, token-based reversible encryption scheme in Ruby that leverages tokenization, HMAC-based secure token mapping, and AES-256-CBC encryption. This approach offers a lightweight yet secure method for text obfuscation, making it a viable alternative for applications where traditional encryption might be overly complex. In the future, I intend to expand the system’s capabilities and further improve its security.

9. References

1. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. https://arxiv.org/abs/1810.04805
   


2. Hugging Face Tokenizers Documentation. https://huggingface.co/docs/tokenizers/python/latest/
   


3. OpenSSL Documentation. https://www.openssl.org/docs/
   


4. Ruby Documentation on OpenSSL. https://ruby-doc.org/stdlib/libdoc/openssl/rdoc/OpenSSL.html