On Apache
Under GNU General Public License
2012-05-24 09:37 @38.107.179.237 Crawled by CCBot/1.0 (+http://www.commoncrawl.org/bot.html)
IO::Compress::Zlib::FAQ(3pm) User Contributed Perl Documentation IO::Compress::Zlib::FAQ(3pm)
NAME
IO::Compress::Zlib::FAQ -- Frequently Asked Questions about IO::Compress::Zlib
DESCRIPTION
Common questions answered.
Compatibility with Unix compress/uncompress.
This module is not compatible with Unix "compress".
If you have the "uncompress" program available, you can use this to read compressed files
open F, "uncompress -c $filename |";
while (<F>)
{
...
Alternatively, if you have the "gunzip" program available, you can use this to read
compressed files
open F, "gunzip -c $filename |";
while (<F>)
{
...
and this to write compress files, if you have the "compress" program available
open F, "| compress -c $filename ";
print F "data";
...
close F ;
Accessing .tar.Z files
See previous FAQ item.
If the "Archive::Tar" module is installed and either the "uncompress" or "gunzip" programs
are available, you can use one of these workarounds to read ".tar.Z" files.
Firstly with "uncompress"
use strict;
use warnings;
use Archive::Tar;
open F, "uncompress -c $filename |";
my $tar = Archive::Tar->new(*F);
...
and this with "gunzip"
use strict;
use warnings;
use Archive::Tar;
open F, "gunzip -c $filename |";
my $tar = Archive::Tar->new(*F);
...
Similarly, if the "compress" program is available, you can use this to write a ".tar.Z"
file
use strict;
use warnings;
use Archive::Tar;
use IO::File;
my $fh = new IO::File "| compress -c >$filename";
my $tar = Archive::Tar->new();
...
$tar->write($fh);
$fh->close ;
Accessing Zip Files
This module provides support for reading/writing zip files using the "IO::Compress::Zip"
and "IO::Uncompress::Unzip" modules.
The primary focus of the "IO::Compress::Zip" and "IO::Uncompress::Unzip" modules is to
provide an "IO::File" compatible streaming read/write interface to zip files/buffers. They
are not fully flegged archivers. If you are looking for an archiver check out the
"Archive::Zip" module. You can find it on CPAN at
http://www.cpan.org/modules/by-module/Archive/Archive-Zip-*.tar.gz
Compressed files and Net::FTP
The "Net::FTP" module provides two low-level methods called "stor" and "retr" that both
return filehandles. These filehandles can used with the "IO::Compress/Uncompress" modules
to compress or uncompress files read from or written to an FTP Server on the fly, without
having to create a temporary file.
Firstly, here is code that uses "retr" to uncompressed a file as it is read from the FTP
Server.
use Net::FTP;
use IO::Uncompress::Gunzip qw(:all);
my $ftp = new Net::FTP ...
my $retr_fh = $ftp->retr($compressed_filename);
gunzip $retr_fh => $outFilename, AutoClose => 1
or die "Cannot uncompress '$compressed_file': $GunzipError\n";
and this to compress a file as it is written to the FTP Server
use Net::FTP;
use IO::Compress::Gzip qw(:all);
my $stor_fh = $ftp->stor($filename);
gzip "filename" => $stor_fh, AutoClose => 1
or die "Cannot compress '$filename': $GzipError\n";
How do I recompress using a different compression?
This is easier that you might expect if you realise that all the "IO::Compress::*" objects
are derived from "IO::File" and that all the "IO::Uncompress::*" modules can read from an
"IO::File" filehandle.
So, for example, say you have a file compressed with gzip that you want to recompress with
bzip2. Here is all that is needed to carry out the recompression.
use IO::Uncompress::Gunzip ':all';
use IO::Compress::Bzip2 ':all';
my $gzipFile = "somefile.gz";
my $bzipFile = "somefile.bz2";
my $gunzip = new IO::Uncompress::Gunzip $gzipFile
or die "Cannot gunzip $gzipFile: $GunzipError\n" ;
bzip2 $gunzip => $bzipFile
or die "Cannot bzip2 to $bzipFile: $Bzip2Error\n" ;
Note, there is a limitation of this technique. Some compression file formats store extra
information along with the compressed data payload. For example, gzip can optionally store
the original filename and Zip stores a lot of information about the original file. If the
original compressed file contains any of this extra information, it will not be
transferred to the new compressed file usign the technique above.
Apache::GZip Revisited
Below is a mod_perl Apache compression module, called "Apache::GZip", taken from
http://perl.apache.org/docs/tutorials/tips/mod_perl_tricks/mod_perl_tricks.html#On_the_Fly_Compression
package Apache::GZip;
#File: Apache::GZip.pm
use strict vars;
use Apache::Constants ':common';
use Compress::Zlib;
use IO::File;
use constant GZIP_MAGIC => 0x1f8b;
use constant OS_MAGIC => 0x03;
sub handler {
my $r = shift;
my ($fh,$gz);
my $file = $r->filename;
return DECLINED unless $fh=IO::File->new($file);
$r->header_out('Content-Encoding'=>'gzip');
$r->send_http_header;
return OK if $r->header_only;
tie *STDOUT,'Apache::GZip',$r;
print($_) while <$fh>;
untie *STDOUT;
return OK;
}
sub TIEHANDLE {
my($class,$r) = @_;
# initialize a deflation stream
my $d = deflateInit(-WindowBits=>-MAX_WBITS()) || return undef;
# gzip header -- don't ask how I found out
$r->print(pack("nccVcc",GZIP_MAGIC,Z_DEFLATED,0,time(),0,OS_MAGIC));
return bless { r => $r,
crc => crc32(undef),
d => $d,
l => 0
},$class;
}
sub PRINT {
my $self = shift;
foreach (@_) {
# deflate the data
my $data = $self->{d}->deflate($_);
$self->{r}->print($data);
# keep track of its length and crc
$self->{l} += length($_);
$self->{crc} = crc32($_,$self->{crc});
}
}
sub DESTROY {
my $self = shift;
# flush the output buffers
my $data = $self->{d}->flush;
$self->{r}->print($data);
# print the CRC and the total length (uncompressed)
$self->{r}->print(pack("LL",@{$self}{qw/crc l/}));
}
1;
Here's the Apache configuration entry you'll need to make use of it. Once set it will
result in everything in the /compressed directory will be compressed automagically.
<Location /compressed>
SetHandler perl-script
PerlHandler Apache::GZip
</Location>
Although at first sight there seems to be quite a lot going on in "Apache::GZip", you
could sum up what the code was doing as follows -- read the contents of the file in
"$r->filename", compress it and write the compressed data to standard output. That's all.
This code has to jump through a few hoops to achieve this because
1. The gzip support in "Compress::Zlib" version 1.x can only work with a real filesystem
filehandle. The filehandles used by Apache modules are not associated with the
filesystem.
2. That means all the gzip support has to be done by hand - in this case by creating a
tied filehandle to deal with creating the gzip header and trailer.
"IO::Compress::Gzip" doesn't have that filehandle limitation (this was one of the reasons
for writing it in the first place). So if "IO::Compress::Gzip" is used instead of
"Compress::Zlib" the whole tied filehandle code can be removed. Here is the rewritten
code.
package Apache::GZip;
use strict vars;
use Apache::Constants ':common';
use IO::Compress::Gzip;
use IO::File;
sub handler {
my $r = shift;
my ($fh,$gz);
my $file = $r->filename;
return DECLINED unless $fh=IO::File->new($file);
$r->header_out('Content-Encoding'=>'gzip');
$r->send_http_header;
return OK if $r->header_only;
my $gz = new IO::Compress::Gzip '-', Minimal => 1
or return DECLINED ;
print $gz $_ while <$fh>;
return OK;
}
or even more succinctly, like this, using a one-shot gzip
package Apache::GZip;
use strict vars;
use Apache::Constants ':common';
use IO::Compress::Gzip qw(gzip);
sub handler {
my $r = shift;
$r->header_out('Content-Encoding'=>'gzip');
$r->send_http_header;
return OK if $r->header_only;
gzip $r->filename => '-', Minimal => 1
or return DECLINED ;
return OK;
}
1;
The use of one-shot "gzip" above just reads from "$r->filename" and writes the compressed
data to standard output.
Note the use of the "Minimal" option in the code above. When using gzip for Content-
Encoding you should always use this option. In the example above it will prevent the
filename being included in the gzip header and make the size of the gzip data stream a
slight bit smaller.
Using "InputLength" to uncompress data embedded in a larger file/buffer.
A fairly common use-case is where compressed data is embedded in a larger file/buffer and
you want to read both.
As an example consider the structure of a zip file. This is a well-defined file format
that mixes both compressed and uncompressed sections of data in a single file.
For the purposes of this discussion you can think of a zip file as sequence of compressed
data streams, each of which is prefixed by an uncompressed local header. The local header
contains information about the compressed data stream, including the name of the
compressed file and, in particular, the length of the compressed data stream.
To illustrate how to use "InputLength" here is a script that walks a zip file and prints
out how many lines are in each compressed file (if you intend write code to walking
through a zip file for real see "Walking through a zip file" in IO::Uncompress::Unzip )
use strict;
use warnings;
use IO::File;
use IO::Uncompress::RawInflate qw(:all);
use constant ZIP_LOCAL_HDR_SIG => 0x04034b50;
use constant ZIP_LOCAL_HDR_LENGTH => 30;
my $file = $ARGV[0] ;
my $fh = new IO::File "<$file"
or die "Cannot open '$file': $!\n";
while (1)
{
my $sig;
my $buffer;
my $x ;
($x = $fh->read($buffer, ZIP_LOCAL_HDR_LENGTH)) == ZIP_LOCAL_HDR_LENGTH
or die "Truncated file: $!\n";
my $signature = unpack ("V", substr($buffer, 0, 4));
last unless $signature == ZIP_LOCAL_HDR_SIG;
# Read Local Header
my $gpFlag = unpack ("v", substr($buffer, 6, 2));
my $compressedMethod = unpack ("v", substr($buffer, 8, 2));
my $compressedLength = unpack ("V", substr($buffer, 18, 4));
my $uncompressedLength = unpack ("V", substr($buffer, 22, 4));
my $filename_length = unpack ("v", substr($buffer, 26, 2));
my $extra_length = unpack ("v", substr($buffer, 28, 2));
my $filename ;
$fh->read($filename, $filename_length) == $filename_length
or die "Truncated file\n";
$fh->read($buffer, $extra_length) == $extra_length
or die "Truncated file\n";
if ($compressedMethod != 8 && $compressedMethod != 0)
{
warn "Skipping file '$filename' - not deflated $compressedMethod\n";
$fh->read($buffer, $compressedLength) == $compressedLength
or die "Truncated file\n";
next;
}
if ($compressedMethod == 0 && $gpFlag & 8 == 8)
{
die "Streamed Stored not supported for '$filename'\n";
}
next if $compressedLength == 0;
# Done reading the Local Header
my $inf = new IO::Uncompress::RawInflate $fh,
Transparent => 1,
InputLength => $compressedLength
or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ;
my $line_count = 0;
while (<$inf>)
{
++ $line_count;
}
print "$filename: $line_count\n";
}
The majority of the code above is concerned with reading the zip local header data. The
code that I want to focus on is at the bottom.
while (1) {
# read local zip header data
# get $filename
# get $compressedLength
my $inf = new IO::Uncompress::RawInflate $fh,
Transparent => 1,
InputLength => $compressedLength
or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ;
my $line_count = 0;
while (<$inf>)
{
++ $line_count;
}
print "$filename: $line_count\n";
}
The call to "IO::Uncompress::RawInflate" creates a new filehandle $inf that can be used to
read from the parent filehandle $fh, uncompressing it as it goes. The use of the
"InputLength" option will guarantee that at most $compressedLength bytes of compressed
data will be read from the $fh filehandle (The only exception is for an error case like a
truncated file or a corrupt data stream).
This means that once RawInflate is finished $fh will be left at the byte directly after
the compressed data stream.
Now consider what the code looks like without "InputLength"
while (1) {
# read local zip header data
# get $filename
# get $compressedLength
# read all the compressed data into $data
read($fh, $data, $compressedLength);
my $inf = new IO::Uncompress::RawInflate \$data,
Transparent => 1,
or die "Cannot uncompress $file [$filename]: $RawInflateError\n" ;
my $line_count = 0;
while (<$inf>)
{
++ $line_count;
}
print "$filename: $line_count\n";
}
The difference here is the addition of the temporary variable $data. This is used to
store a copy of the compressed data while it is being uncompressed.
If you know that $compressedLength isn't that big then using temporary storage won't be a
problem. But if $compressedLength is very large or you are writing an application that
other people will use, and so have no idea how big $compressedLength will be, it could be
an issue.
Using "InputLength" avoids the use of temporary storage and means the application can cope
with large compressed data streams.
One final point -- obviously "InputLength" can only be used whenever you know the length
of the compressed data beforehand, like here with a zip file.
SEE ALSO
Compress::Zlib, IO::Compress::Gzip, IO::Uncompress::Gunzip, IO::Compress::Deflate,
IO::Uncompress::Inflate, IO::Compress::RawDeflate, IO::Uncompress::RawInflate,
IO::Compress::Bzip2, IO::Uncompress::Bunzip2, IO::Compress::Lzop, IO::Uncompress::UnLzop,
IO::Compress::Lzf, IO::Uncompress::UnLzf, IO::Uncompress::AnyInflate,
IO::Uncompress::AnyUncompress
Compress::Zlib::FAQ
File::GlobMapper, Archive::Zip, Archive::Tar, IO::Zlib
AUTHOR
This module was written by Paul Marquess, pmqs AT cpan.org.
MODIFICATION HISTORY
See the Changes file.
COPYRIGHT AND LICENSE
Copyright (c) 2005-2008 Paul Marquess. All rights reserved.
This program is free software; you can redistribute it and/or modify it under the same
terms as Perl itself.
perl v5.10.0 2008-07-15 IO::Compress::Zlib::FAQ(3pm)
Generated by $Id: phpMan.php,v 4.49 2006/02/26 13:18:18 chedong Exp $ Author: Che Dong
On Apache
Under GNU General Public License
2012-05-24 09:37 @38.107.179.237 Crawled by CCBot/1.0 (+http://www.commoncrawl.org/bot.html)