Difference between pipe and stream in node.js

Here's my code

const fs = require('fs');
const src = fs.createReadStream('bigfile3.txt');
const des = fs.createWriteStream('newTest.txt');

I can use either

src.on('data',(chunk)=>{
des.write(chunk);});

Or

src.pipe(des);

Is there any difference between this two ways of handling the file operation? The pipe method gives me an error of > "size" argument must not be larger than 2147483647 whenever I try with a large file.(~2GB)

Can anyone explain the working behind pipe and stream? Thanks.

Answers:

Answer

You should use the pipe method because the flow of data will be automatically managed so that the destination Writable stream is not overwhelmed by a faster Readable stream.

If your readable stream is faster than the writable stream then you may experience data loss in des.write(data) method so better you should use src.pipe(des);

If the file size is big then you should use streams, thats the correct way of doing it, I tried similar example like yours to copy 3.5 GB file with streams and pipe, it worked flawlessly in my case. Check you must be doing something wrong.

The example which I tried

'use strict'

const fs =require('fs')
const readStream = fs.createReadStream('./Archive.zip')
const writeStream = fs.createWriteStream('./Archive3.zip')


readStream.pipe(writeStream)

Docs:

https://nodejs.org/api/stream.html#stream_writable_write_chunk_encoding_callback

https://nodejs.org/api/stream.html#stream_readable_pipe_destination_options

Tags

Recent Questions

Top Questions

Home Tags Terms of Service Privacy Policy DMCA Contact Us

©2020 All rights reserved.