In today’s world, secure file transfer between servers is a critically important task. Typically, SFTP or SCP protocols are used for this purpose. While SFTP offers advanced features and is considered more modern, its use may be prohibited in some systems due to security or compatibility reasons. In such cases, the old but time-tested SCP protocol comes to the rescue.
Analyzing existing solutions for working with the ssh2 package in Node.js, it becomes apparent that many of them use the SFTP subsystem to implement SCP. However, SCP and SFTP are two different protocols. Unlike SFTP, SCP does not support interactive mode and cannot process command scripts, which means all commands must be passed directly through the command line.
Moreover, even though both SCP and SFTP use the same SSH encryption for file transfer with similar overhead, SCP usually operates significantly faster when transferring files, especially in high-latency networks. This is because SCP implements a more efficient transfer algorithm that does not require waiting for packet acknowledgments. This approach increases transfer speed but does not allow interrupting the process without terminating the entire session, unlike SFTP.
In this article, we will explore how to set up and use SCP with the ssh2 package in your Node.js applications. This is particularly relevant in situations where SFTP is unavailable, but secure file transfer via the SCP protocol is required.
How SCP Works and Practical Examples
The SCP protocol is used for secure file transfer between local and remote hosts over SSH. Unlike SFTP, SCP does not support interactive commands and operates on the principle of message exchange in a strictly defined order. Understanding this sequence is crucial when implementing SCP using the ssh2 package in Node.js.
When transferring a file from a remote host, the client and server exchange messages as follows:
1) Establishing Connection and Authorization: First, we establish an SSH connection with the remote server and perform authorization.
const connection = new Client();
...
connection.connect(connectionOptions);
2) Sending the SCP Command: Upon receiving the ready event, the client sends the scp command with the necessary options to the server via the exec channel.
connection.exec(`scp -f ${remoteFile}`, (err, readStream) => {...});
3) Initializing the Transfer: To prompt SCP to send a response, you must first send the initial readiness signal 0x00 to synchronize actions between the client and server.
const { Client } = require('ssh2');
const fs = require('fs');
const remoteFile = '/opt/test.tar.gz';
const connection = new Client();
connection.on('ready', () => {
connection.exec(`scp -f ${remoteFile}`, (err, readStream) => {
if (err) {
console.error(err);
connection.end();
return;
}
// Send the initial acknowledgment byte
readStream.write(Buffer.from([0]));
});
}).connect({
host: 'your_remote_host',
port: 22,
username: 'your_username',
privateKey: fs.readFileSync('/path/to/your/private/key')
});
After sending the initial 0x00 byte, the server will start transmitting the file’s metadata and its contents. We need to process this data and save the file on the local machine.
Let’s add event handlers for readStream to properly process incoming data and save the file.
const { Client } = require('ssh2');
const fs = require('fs');
const path = require('path');
const remoteFile = '/opt/test.tar.gz';
const localFile = path.basename(remoteFile); // Local filename for saving
const connection = new Client();
connection.on('ready', () => {
connection.exec(`scp -f ${remoteFile}`, (err, stream) => {
if (err) {
console.error('Error executing SCP command:', err);
connection.end();
return;
}
let fileStream;
let fileSize = 0;
let receivedBytes = 0;
let expect = 'response'; // Current expected state
let buffer = Buffer.alloc(0);
// Function to send acknowledgment
const sendByte = (byte) => {
stream.write(Buffer.from([byte]));
};
// Send the initial acknowledgment byte
sendByte(0);
stream.on('data', (data) => {
buffer = Buffer.concat([buffer, data]);
while (true) {
if (buffer.length < 1) break;
if (expect === 'response') {
const response = buffer[0];
if (response === 0x43) {
expect = 'metadata';
} else if (response === 0x01 || response === 0x02) {
expect = 'error';
buffer = buffer.slice(1);
} else {
console.error('Unknown server response:', response.toString(16));
connection.end();
return;
}
} else if (expect === 'error') {
const nlIndex = buffer.indexOf(0x0A); // '\n'
if (nlIndex === -1) break; // Waiting for more data
const errorMsg = buffer.slice(0, nlIndex).toString();
console.error(`SCP Error: ${errorMsg}`);
buffer = buffer.slice(nlIndex + 1);
connection.end();
return;
} else if (expect === 'metadata') {
const nlIndex = buffer.indexOf(0x0A); // '\n'
if (nlIndex === -1) break; // Waiting for more data
const metadata = buffer.slice(0, nlIndex).toString();
buffer = buffer.slice(nlIndex + 1);
if (metadata.startsWith('C')) {
const parts = metadata.split(' ');
if (parts.length < 3) {
console.error('Error: Invalid metadata:', metadata);
connection.end();
return;
}
fileSize = parseInt(parts[1], 10);
if (isNaN(fileSize)) {
console.error('Error: Invalid file size in metadata:', metadata);
connection.end();
return;
}
// Create a write stream for the file
fileStream = fs.createWriteStream(localFile);
fileStream.on('error', (fileErr) => {
console.error('Error writing file:', fileErr);
connection.end();
});
// Send acknowledgment
sendByte(0);
expect = 'data';
receivedBytes = 0;
console.log('Starting file transfer...');
} else {
console.error('Error: Expected metadata line, received:', metadata);
connection.end();
return;
}
} else if (expect === 'data') {
if (receivedBytes < fileSize) {
const remainingBytes = fileSize - receivedBytes;
const bytesToRead = Math.min(buffer.length, remainingBytes);
fileStream.write(buffer.slice(0, bytesToRead));
receivedBytes += bytesToRead;
buffer = buffer.slice(bytesToRead);
if (receivedBytes === fileSize) {
expect = 'data_response';
}
} else {
expect = 'data_response';
}
} else if (expect === 'data_response') {
console.log('Bytes received:', receivedBytes);
if (buffer.length < 1) break; // Waiting for more data
const response = buffer[0];
buffer = buffer.slice(1);
if (response === 0) {
fileStream.end(() => {
console.log(`File ${localFile} saved successfully.`);
});
expect = 'end';
sendByte(0); // Send acknowledgment to finish
} else if (response === 1 || response === 2) {
expect = 'error';
} else {
console.error('Unknown server response after data transfer:', response);
connection.end();
return;
}
} else if (expect === 'end') {
// Transfer completed
connection.end();
return;
}
}
});
stream.on('close', () => {
console.log('Transfer completed.');
connection.end();
});
stream.stderr.on('data', (data) => {
console.error(`STDERR: ${data.toString()}`);
});
});
}).connect({
host: 'your_remote_host',
port: 22,
username: 'your_username',
privateKey: fs.readFileSync('/path/to/your/private/key')
});
The most important thing to consider is sending the 0x00 byte at the right moments to ensure correct synchronization with the server.
In this example, I implemented a basic file transfer mechanism using SCP. Note that this code does not include error handlers, which you will need to add yourself.
SOCIAL SHARE CARD GENERATOR