Processing the file as its being uploaded

[Marcelo Taube]

Hello,
I want to write a php script which process a file being upload to the
server, to check contentes and stores it on the fly, as its being uploaded.
My goal is to avoid using the memory needed to hold the whole file, and
instead just upload it and forgetting the parts already processed.
I have seen that this "streaming" interface of file uploads exist for HTTP
PUT method but could not find info on how to do it using POST.
So, is there any way of configuring PHP or APACHE to provide uploaded files
using POST in a streaming fashion?
If this is not possible, is it because an instrinsic limit of the HTTP
protocol? a limit on the apache architecture? or a design decision in PHP?
And last, in PUT method, are I warrantied that the input in stdin comes
directly to the PHP script or should i expect apache to pre buffer the
whole file and then just start to send it?

Thank you

[Jose Nobile]

Hi,

Running PHP as Apache module, not possible until file is uploaded. Perhaps
using cgi. The reason is the architecture in PHP, you don't have "events"
in php, is not possible start the script, and attach an event when file is
uploaded. In perl is possible.

But, don't worry about memory, file is saved directly in HDD, from tmp
folder you can read in chunks or in any way, streams, etc.


Saludos,
José Nobile

Post by Marcelo Taube
Hello,
I want to write a php script which process a file being upload to the
server, to check contentes and stores it on the fly, as its being uploaded.
My goal is to avoid using the memory needed to hold the whole file, and
instead just upload it and forgetting the parts already processed.
I have seen that this "streaming" interface of file uploads exist for HTTP
PUT method but could not find info on how to do it using POST.
So, is there any way of configuring PHP or APACHE to provide uploaded files
using POST in a streaming fashion?
If this is not possible, is it because an instrinsic limit of the HTTP
protocol? a limit on the apache architecture? or a design decision in PHP?
And last, in PUT method, are I warrantied that the input in stdin comes
directly to the PHP script or should i expect apache to pre buffer the
whole file and then just start to send it?
Thank you

[Marcelo Taube]

Thanks jose
I prefer not to have temporary files nor files in memory.
I wonder if the PUT method achieves my desired results though.

Post by Jose Nobile
Hi,
Running PHP as Apache module, not possible until file is uploaded. Perhaps
using cgi. The reason is the architecture in PHP, you don't have "events"
in php, is not possible start the script, and attach an event when file is
uploaded. In perl is possible.
But, don't worry about memory, file is saved directly in HDD, from tmp
folder you can read in chunks or in any way, streams, etc.
Saludos,
José Nobile

Post by Marcelo Taube
Hello,
I want to write a php script which process a file being upload to the
server, to check contentes and stores it on the fly, as its being uploaded.
My goal is to avoid using the memory needed to hold the whole file, and
instead just upload it and forgetting the parts already processed.
I have seen that this "streaming" interface of file uploads exist for HTTP
PUT method but could not find info on how to do it using POST.
So, is there any way of configuring PHP or APACHE to provide uploaded files
using POST in a streaming fashion?
If this is not possible, is it because an instrinsic limit of the HTTP
protocol? a limit on the apache architecture? or a design decision in PHP?
And last, in PUT method, are I warrantied that the input in stdin comes
directly to the PHP script or should i expect apache to pre buffer the
whole file and then just start to send it?
Thank you

[Jim Giner]

Post by Marcelo Taube
Thanks jose
I prefer not to have temporary files nor files in memory.
I wonder if the PUT method achieves my desired results though.

Forgive my stupidity, but what is the PUT method? I've never heard of
that.

As for temporary files - that is how the http upload does it - you use
php to capture that temp file and move it to its intended destination.
You could instead just read the temp file and do your thing and then
discard it or whatever. What's the problem with that? Are these files
so huge that you are afraid of using your memory allotment? If so- I'd
worry about the upload step first.

[Robert Williams]

On Nov 28, 2013, at 13:07, "Jim Giner" <***@albanyhandball.com<mailto:***@albanyhandball.com>> wrote:

Forgive my stupidity, but what is the PUT method? I've never heard of
that.

PUT is one of the HTTP 1.1 methods:

<http://tools.ietf.org/html/rfc2616>

Or more specifically:

<http://tools.ietf.org/html/rfc2616#section-9.6>

You're certainly already aware of the POST and GET methods, but there are several others, of which PUT is one. You'll most often see the others at work in REST-based web services.

[Camilo Sperberg]

Post by Marcelo Taube
Hello,
I want to write a php script which process a file being upload to the
server, to check contentes and stores it on the fly, as its being uploaded.
My goal is to avoid using the memory needed to hold the whole file, and
instead just upload it and forgetting the parts already processed.
I have seen that this "streaming" interface of file uploads exist for HTTP
PUT method but could not find info on how to do it using POST.
So, is there any way of configuring PHP or APACHE to provide uploaded files
using POST in a streaming fashion?
If this is not possible, is it because an instrinsic limit of the HTTP
protocol? a limit on the apache architecture? or a design decision in PHP?
And last, in PUT method, are I warrantied that the input in stdin comes
directly to the PHP script or should i expect apache to pre buffer the
whole file and then just start to send it?
Thank you

We had recently a look into plupload which enables the upload (via JavaScript) of chunks of data, which, when completed are just joined together (PHP's side). It enables you to not increase memory usage because of one large upload.

You can find demos and documentation on their github repo:
https://github.com/moxiecode/plupload

It hasn't made it into production yet but on our tests it works pretty well, IE11 has a few problems, but there is a fix for that which should make it into the next stable version.
https://github.com/moxiecode/plupload/issues/906

Some more information about chunking and plupload:
https://github.com/moxiecode/plupload/wiki/Frequently-Asked-Questions#wiki-when-to-use-chunking-and-when-not

Greetings.

Met vriendelijke groet,
Camilo Sperberg

----------------
W: http://unreal4u.com
T: http://twitter.com/unreal4u