libzypp: zypp::Fetcher Class Reference

libzypp 17.38.11

zypp::Fetcher Class Reference

This class allows to retrieve a group of files in a confortable way, providing some smartness that does not belong to the media layer like: More...

`#include <zypp/Fetcher.h>`

## Classes
class | Impl
| Fetcher implementation. More...

## Public Types
enum | Option { AutoAddContentFileIndexes = 0x0001 , AutoAddChecksumsIndexes = 0x0002 , AutoAddIndexes = AutoAddContentFileIndexes | AutoAddChecksumsIndexes }
| Various option flags to change behavior. More...
enum | CacheOptions { Default = 0 , CleanFiles = 1 }

## Public Member Functions
| ZYPP_DECLARE_FLAGS (Options, Option)
| Fetcher ()
| Default ctor.
virtual | ~Fetcher ()
| Dtor.
void | setOptions (Options options)
| Set the Fetcher options.
Options | options () const
| Get current options.
void | addIndex (const OnMediaLocation &resource)
| Adds an index containing metadata (for example checksums ) that will be retrieved and read before the job processing starts.
void | enqueue (const OnMediaLocation &resource, const FileChecker &checker=FileChecker())
| Enqueue a object for transferal, they will not be transferred until start() is called.
void | enqueueDigested (const OnMediaLocation &resource, const FileChecker &checker=FileChecker())
| Enqueue a object for transferal, they will not be transferred until start() is called.
ZYPP_DEPRECATED void | enqueueDigested (const OnMediaLocation &resource, const FileChecker &checker, const Pathname &deltafile)
void | enqueueDir (const OnMediaLocation &resource, bool recursive=false, const FileChecker &checker=FileChecker())
| Enqueue a directory.
void | enqueueDigestedDir (const OnMediaLocation &resource, bool recursive=false, const FileChecker &checker=FileChecker())
| Enqueue a directory and always check for checksums.
void | addCachePath (const Pathname &cache_dir)
| adds a directory to the list of directories where to look for cached files
void | addCachePath (const Pathname &cache_dir, CacheOptions options)
| adds a directory to the list of directories where to look for cached files and options on how to handle the cache
void | reset ()
| Reset the transfer (jobs) list.
void | setMediaSetAccess (MediaSetAccess &media)
| Sets the media set access that will be used to precache and to download the files when start is called.
void | start (const Pathname &dest_dir, const ProgressData::ReceiverFnc &progress=ProgressData::ReceiverFnc())
| start the transfer to a destination directory dest_dir The media has to be provides with setMediaSetAccess before calling this version of start.
void | start (const Pathname &dest_dir, MediaSetAccess &media, const ProgressData::ReceiverFnc &progress=ProgressData::ReceiverFnc())
| start the transfer to a destination directory dest_dir You have to provde a media set access media to get the files from The file tree will be replicated inside this directory

## Static Public Member Functions
static Pathname | mapToCachePath (Pathname remotePath_r)
| Map a resource filename to a local path below a destDir.
static Pathname | mapToCachePath (const OnMediaLocation &resource_r)

## Private Attributes
RWCOW_pointer< Impl > | _pimpl
| Pointer to implementation.

## Friends
std::ostream & | operator<< (std::ostream &str, const Fetcher &obj)
| relates: Fetcher Stream output

## Detailed Description

This class allows to retrieve a group of files in a confortable way, providing some smartness that does not belong to the media layer like:

- Configurable local caches to retrieve already donwloaded files.
- File checkers that can check for right checksums digital signatures, etc.

MediaSetAccess access(url, path);
Fetcher fetcher;
fetcher.enqueue( OnMediaLocation().setLocation("/somefile") );
fetcher.addCachePath("/tmp/cache")
fetcher.start( "/download-dir, access );
fetcher.reset();
zypp::Fetcher::start
void start(const Pathname &dest_dir, const ProgressData::ReceiverFnc &progress=ProgressData::ReceiverFnc())
start the transfer to a destination directory dest_dir The media has to be provides with setMediaSetA...
Definition Fetcher.cc:978
zypp::Fetcher::enqueue
void enqueue(const OnMediaLocation &resource, const FileChecker &checker=FileChecker())
Enqueue a object for transferal, they will not be transferred until start() is called.
Definition Fetcher.cc:952
zypp::Fetcher::Fetcher
Fetcher()
Default ctor.
Definition Fetcher.cc:904
zypp::Fetcher::addCachePath
void addCachePath(const Pathname &cache_dir)
adds a directory to the list of directories where to look for cached files
Definition Fetcher.cc:958
zypp::MediaSetAccess
Media access layer responsible for handling files distributed on a set of media with media change and...
Definition MediaSetAccess.h:82
zypp::OnMediaLocation
Describes a resource file located on a medium.
Definition onmedialocation.h:37
zypp::url
Url details namespace.
Definition UrlBase.cc:58

To use the checkers. just create a functor implementing bool operator()(const Pathname &file)

See also
  FileChecker. Pass the necessary validation data in the constructor of the functor, and pass the object to the enqueue method.

ChecksumFileChecker checker(CheckSum("sha1", "....");
fetcher.enqueue( location, checker);
zypp::CheckSum
Definition CheckSum.h:34
zypp::ChecksumFileChecker
Built in file checkers.
Definition FileChecker.h:48

If you need to use more than one checker

See also
  CompositeFileChecker

Additionally, you can automatically enqueue a job with a checksum checker by using enqueueDigested which will use the OnMediaLocation checksum automatically.

location.setChecksum(CheckSum("sha1", "...."));
fetcher.enqueueDigested(location);
zypp::Fetcher::enqueueDigested
void enqueueDigested(const OnMediaLocation &resource, const FileChecker &checker=FileChecker())
Enqueue a object for transferal, they will not be transferred until start() is called.
Definition Fetcher.cc:926

Note
  If the checksum of the location is empty, but enqueueDigested is used, then the user will get a warning that the file has no checksum.

Additionally, Fetcher supports checking the downloaded content by using signed indexes on the remote side.

MediaSetAccess access(url, path);
Fetcher fetcher;
fetcher.addIndex(OnMediaLocation("/content"));
fetcher.enqueue( OnMediaLocation().setLocation("/somefile") );
fetcher.start( "/download-dir, access );
fetcher.reset();
zypp::Fetcher::addIndex
void addIndex(const OnMediaLocation &resource)
Adds an index containing metadata (for example checksums ) that will be retrieved and read before the...
Definition Fetcher.cc:946

Indexes are supported in CHECKSUMS format (simple text file) with checksum and file name, or content file, whith HASH SHA1 line entries.

Note
  The indexes file names are relative to the directory where the index is.
  libzypp-11.x: Introduction of sha256 lead to the insight that using SHA1SUMS as filename was a bad choice. New media store the checksums in a CHECKSUMS file. If a CHECKSUMS file is not present, we fall back looking for a SHA1SUMS file. The checksum type (md5,sha1,sha256) is auto detected by looking at the cheksums length. No need to somehow encode it in the filename.
  bsc#1253740: Replicating the file tree below the destination directory fails if filenames stored in an OnMediaLocation are relative and referring to "../". They will point to a location outside the cache. To fix the Fetcher writing to a location outside the destination directory all remote paths will be mapped to filenames below the destination directory. mapToCachePath will return the location below the destination directory, where a remote resource will be located.

Definition at line 111 of file Fetcher.h.

## Member Enumeration Documentation

## ◆ Option

enum zypp::Fetcher::Option

Various option flags to change behavior.

Enumerator
AutoAddContentFileIndexes | If a content file is found, it is downloaded and read.
AutoAddChecksumsIndexes | If a CHECKSUMS file is found, it is downloaded and read.
AutoAddIndexes | If a content or CHECKSUMS file is found, it is downloaded and read.

Definition at line 123 of file Fetcher.h.

## ◆ CacheOptions

enum zypp::Fetcher::CacheOptions

Enumerator
Default |
CleanFiles |

Definition at line 313 of file Fetcher.h.

## Constructor & Destructor Documentation

## ◆ Fetcher()

zypp::Fetcher::Fetcher | ( |  | ) |

Default ctor.

Definition at line 904 of file Fetcher.cc.

## ◆ ~Fetcher()

zypp::Fetcher::~Fetcher | ( |  | ) |
| virtual

Dtor.

Definition at line 908 of file Fetcher.cc.

## Member Function Documentation

## ◆ ZYPP_DECLARE_FLAGS()

zypp::Fetcher::ZYPP_DECLARE_FLAGS | ( | Options | ,
|  | Option | )

## ◆ setOptions()

void zypp::Fetcher::setOptions | ( | Options | options | ) |

Set the Fetcher options.

See also
  Fetcher::Options

Definition at line 911 of file Fetcher.cc.

## ◆ options()

Fetcher::Options zypp::Fetcher::options | ( |  | ) | const

Get current options.

See also
  Fetcher::Options

Definition at line 916 of file Fetcher.cc.

## ◆ addIndex()

void zypp::Fetcher::addIndex | ( | const OnMediaLocation & | resource | ) |

Adds an index containing metadata (for example checksums ) that will be retrieved and read before the job processing starts.

Nothing will be transferred or checked until start() is called.

The index is relative to the media path, and the listed files too.

Indexes in the SHA1SUM format, and YaST content file

The file has to be signed or the user will be warned that the file is unsigned. You can place the signature next to the file adding the .asc extension.

If you expect the key to not to be in the target system, then you can place it next to the index using adding the .key extension.

Definition at line 946 of file Fetcher.cc.

## ◆ enqueue()

void zypp::Fetcher::enqueue | ( | const OnMediaLocation & | resource,
|  | const FileChecker & | checker = FileChecker() )

Enqueue a object for transferal, they will not be transferred until start() is called.

Definition at line 952 of file Fetcher.cc.

## ◆ enqueueDigested() [1/2]

void zypp::Fetcher::enqueueDigested | ( | const OnMediaLocation & | resource,
|  | const FileChecker & | checker = FileChecker() )

Enqueue a object for transferal, they will not be transferred until start() is called.

Note
  As OnMediaLocation contains the digest information, a ChecksumFileChecker is automatically added to the transfer job, so make sure you don't add another one or the user could be asked twice.

Todo
  FIXME implement checker == operator to avoid this.

Definition at line 926 of file Fetcher.cc.

## ◆ enqueueDigested() [2/2]

void zypp::Fetcher::enqueueDigested | ( | const OnMediaLocation & | resource,
|  | const FileChecker & | checker,
|  | const Pathname & | deltafile )

Deprecated
  the deltafile argument is part of OnMediaLocation now, please use enqueueDigested without the deltafile argument

Definition at line 921 of file Fetcher.cc.

## ◆ enqueueDir()

void zypp::Fetcher::enqueueDir | ( | const OnMediaLocation & | resource,
|  | bool | recursive = false,
|  | const FileChecker & | checker = FileChecker() )

Enqueue a directory.

As the files to be enqueued are not known in advance, all files whose checksum can be found in some index are enqueued with checksum checking. Otherwise they are not.

Some index may provide the checksums, either by addIndex or using AutoAddIndexes flag.

Files are checked by providing a CHECKSUMS or content file listing <checksum> filename and a respective CHECKSUMS.asc/content.asc which has the signature for the checksums.

If you expect the user to not have the key of the signature either in the trusted or untrusted keyring, you can offer it as CHECKSUMS.key (or content.key)

Parameters

  recursive | True if the complete tree should be enqueued.

Note
  As checksums are read from the index, a ChecksumFileChecker is automatically added to transfer jobs having a checksum available, so make sure you don't add another one or the user could be asked twice.
  The format of the file CHECKSUMS is the output of: ls | grep -v CHECKSUMS | xargs sha256sum > CHECKSUMS in each subdirectory.
  Every file CHECKSUMS.* except of CHECKSUMS.(asc|key|(void)) will not be transferred and will be ignored.

Definition at line 931 of file Fetcher.cc.

## ◆ enqueueDigestedDir()

void zypp::Fetcher::enqueueDigestedDir | ( | const OnMediaLocation & | resource,
|  | bool | recursive = false,
|  | const FileChecker & | checker = FileChecker() )

Enqueue a directory and always check for checksums.

As the files to be enqueued are not known in advance, all files are enqueued with checksum checking. If the checksum of some file is not in some index, then there will be a verification warning ( DigestReport ).

Therefore some index will need to provide the checksums, either by addIndex or using AutoAddIndexes flag.

Files are checked by providing a CHECKSUMS or content file listing <checksum> filename and a respective CHECKSUMS.asc/content.asc which has the signature for the checksums.

If you expect the user to not have the key of the signature either in the trusted or untrusted keyring, you can offer it as CHECKSUMS.key (or content.key)

Parameters

  recursive | True if the complete tree should be enqueued.

Note
  As checksums are read from the index, a ChecksumFileChecker is automatically added to every transfer job, so make sure you don't add another one or the user could be asked twice.
  The format of the file CHECKSUMS is the output of: ls | grep -v CHECKSUMS | xargs sha256sum > CHECKSUMS in each subdirectory.
  Every file CHECKSUMS.* except of CHECKSUMS.(asc|key|(void)) will not be transferred and will be ignored.

Definition at line 938 of file Fetcher.cc.

## ◆ addCachePath() [1/2]

void zypp::Fetcher::addCachePath | ( | const Pathname & | cache_dir | ) |

adds a directory to the list of directories where to look for cached files

Definition at line 958 of file Fetcher.cc.

## ◆ addCachePath() [2/2]

void zypp::Fetcher::addCachePath | ( | const Pathname & | cache_dir,
|  | CacheOptions | options )

adds a directory to the list of directories where to look for cached files and options on how to handle the cache

Definition at line 963 of file Fetcher.cc.

## ◆ reset()

void zypp::Fetcher::reset | ( |  | ) |

Reset the transfer (jobs) list.

Note
  It does not reset the cache directory list

Definition at line 968 of file Fetcher.cc.

## ◆ setMediaSetAccess()

void zypp::Fetcher::setMediaSetAccess | ( | MediaSetAccess & | media | ) |

Sets the media set access that will be used to precache and to download the files when start is called.

Calling start will reset the media access.

Definition at line 973 of file Fetcher.cc.

## ◆ start() [1/2]

void zypp::Fetcher::start | ( | const Pathname & | dest_dir,
|  | const ProgressData::ReceiverFnc & | progress = ProgressData::ReceiverFnc() )

start the transfer to a destination directory dest_dir The media has to be provides with setMediaSetAccess before calling this version of start.

The file tree will be replicated inside this directory

Definition at line 978 of file Fetcher.cc.

## ◆ start() [2/2]

void zypp::Fetcher::start | ( | const Pathname & | dest_dir,
|  | MediaSetAccess & | media,
|  | const ProgressData::ReceiverFnc & | progress = ProgressData::ReceiverFnc() )

start the transfer to a destination directory dest_dir You have to provde a media set access media to get the files from The file tree will be replicated inside this directory

Definition at line 983 of file Fetcher.cc.

## ◆ mapToCachePath() [1/2]

Pathname zypp::Fetcher::mapToCachePath | ( | Pathname | remotePath_r | ) |
| static

Map a resource filename to a local path below a destDir.

bsc#1253740: To fix the Fetcher writing to a location outside the destination directory if remote paths have leading "../"s, this function maps a resource filename to it's location below the destination directory.

Note
  It's an implementation detail, that in many cases the "../"s are encoded as "%2E%2E/"s. This may change in future version if necessary.

Definition at line 990 of file Fetcher.cc.

## ◆ mapToCachePath() [2/2]

Pathname zypp::Fetcher::mapToCachePath | ( | const OnMediaLocation & | resource_r | ) |
| inlinestatic

Definition at line 372 of file Fetcher.h.

## Friends And Related Symbol Documentation

## ◆ operator<<

std::ostream & operator<< | ( | std::ostream & | str,
|  | const Fetcher & | obj )
| friend

relates: Fetcher Stream output

Definition at line 996 of file Fetcher.cc.

## Member Data Documentation

## ◆ _pimpl

RWCOW_pointer<Impl> zypp::Fetcher::_pimpl
| private

Pointer to implementation.

Definition at line 377 of file Fetcher.h.

The documentation for this class was generated from the following files:

- zypp/Fetcher.h
- zypp/Fetcher.cc