Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide way to get image size without using extents #67

Open
ahadas opened this issue May 11, 2022 · 1 comment
Open

Provide way to get image size without using extents #67

ahadas opened this issue May 11, 2022 · 1 comment
Labels
enhancement Enhancing the system by adding new feature or improving performance or reliability performance

Comments

@ahadas
Copy link
Member

ahadas commented May 11, 2022

The http backed need to know the image size for validating requests to the
server. Since we did not have a way to get the size from the server, we
implemented size() using extents request, and cached the extents for future
calls.

Using extents to get the size is inefficient, and may take lot of time on
large images on fragmented file systems.

Add the image size the OPTIONS response:

Example request:

OPTIONS /images/tiket-id

Return json with:

{
   "features": ["flush", "zero", "extents"],
   "unix_socket": "...",
   "max_readers": 8,
   "max_writers": 8,
   "size": 53687091200,
   ... 
}

The size should be the effective size that can downloaded or uploaded,
using the same logic used to validate range requests.

I think we need to add more info that can be helpful for the client, for
example:

  • the minimum block size (we want aligned writes)
  • the preferred request size (128m?)
  • transfer id (used for debugging transfer flows)
  • sparse (writing zero will deacllocate space)
  • dirty (dirty extents available, only during backup)

This maps nicely to the NBD protocol handshake, reporting all he info about
the remote disk.

The http backend should be updated to use the size reported in OPTIONS
instead of doing a slow extent request to get the size.

Original bug: https://bugzilla.redhat.com/1924945

@ahadas ahadas added the good first issue Good issue for a new contributor label May 11, 2022
@nirs nirs added enhancement Enhancing the system by adding new feature or improving performance or reliability performance and removed good first issue Good issue for a new contributor labels May 15, 2022
@nirs
Copy link
Member

nirs commented May 24, 2022

This is also helps to avoid disconnection of idle connections (#71) if a client is using multiple connections and is waiting util the initial EXTENTS request completes.

In imageio client this does not happen since we send the first EXTENTS request before creating the other connections, as part of initial validation. But other clients may use different design and hit this issue.

aesteve-rh pushed a commit to nirs/ovirt-imageio that referenced this issue Jul 22, 2022
Report image virtual size in OPTIONS so clients can get the image size
without a possibly slow extent call.

Report the format since OPTIONS can report the virtual size only when
the backend provides raw format. This is used when using the http
backend to report OPTIONS to the client.

Reporting virtual size is easy with the nbd and memory backends since
they always use raw format.

When using file backend and qcow2 image, we don't have access to the
image virtual size, and this size is not helpful to the user uploading
or downloading data. Currently we don't know about the image format
since engine does not report it in the ticket.

The http backend reports the info from the remote server, so it depends
on the backend used by the remote server, and on having new server
reporting the format and size.

To keep code and the API simple, we report virtual size only when using
the nbd and memory backends. When engine will report the image format
for the file backend, we can also report the size for raw images access
via the file backend.

Change-Id: I89118301c98dc2d11c25a4d1e7ef83df26336f01
Related: oVirt#67
Bug-Url: https://bugzilla.redhat.com/1924945
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
aesteve-rh pushed a commit to nirs/ovirt-imageio that referenced this issue Jul 29, 2022
Report image virtual size in OPTIONS so clients can get the image size
without a possibly slow extent call.

Report the format since OPTIONS can report the virtual size only when
the backend provides raw format. This is used when using the http
backend to report OPTIONS to the client.

Reporting virtual size is easy with the nbd and memory backends since
they always use raw format.

When using file backend and qcow2 image, we don't have access to the
image virtual size, and this size is not helpful to the user uploading
or downloading data. Currently we don't know about the image format
since engine does not report it in the ticket.

The http backend reports the info from the remote server, so it depends
on the backend used by the remote server, and on having new server
reporting the format and size.

To keep code and the API simple, we report virtual size only when using
the nbd and memory backends. When engine will report the image format
for the file backend, we can also report the size for raw images access
via the file backend.

Change-Id: I89118301c98dc2d11c25a4d1e7ef83df26336f01
Related: oVirt#67
Bug-Url: https://bugzilla.redhat.com/1924945
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
aesteve-rh pushed a commit to nirs/ovirt-imageio that referenced this issue Jul 29, 2022
Report image virtual size in OPTIONS so clients can get the image size
without a possibly slow extent call.

Report the format since OPTIONS can report the virtual size only when
the backend provides raw format. This is used when using the http
backend to report OPTIONS to the client.

Reporting virtual size is easy with the nbd and memory backends since
they always use raw format.

When using file backend and qcow2 image, we don't have access to the
image virtual size, and this size is not helpful to the user uploading
or downloading data. Currently we don't know about the image format
since engine does not report it in the ticket.

The http backend reports the info from the remote server, so it depends
on the backend used by the remote server, and on having new server
reporting the format and size.

To keep code and the API simple, we report virtual size only when using
the nbd and memory backends. When engine will report the image format
for the file backend, we can also report the size for raw images access
via the file backend.

Change-Id: I89118301c98dc2d11c25a4d1e7ef83df26336f01
Related: oVirt#67
Bug-Url: https://bugzilla.redhat.com/1924945
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
aesteve-rh pushed a commit to nirs/ovirt-imageio that referenced this issue Aug 8, 2022
Report image virtual size in OPTIONS so clients can get the image size
without a possibly slow extent call.

Report the format since OPTIONS can report the virtual size only when
the backend provides raw format. This is used when using the http
backend to report OPTIONS to the client.

Reporting virtual size is easy with the nbd and memory backends since
they always use raw format.

When using file backend and qcow2 image, we don't have access to the
image virtual size, and this size is not helpful to the user uploading
or downloading data. Currently we don't know about the image format
since engine does not report it in the ticket.

The http backend reports the info from the remote server, so it depends
on the backend used by the remote server, and on having new server
reporting the format and size.

To keep code and the API simple, we report virtual size only when using
the nbd and memory backends. When engine will report the image format
for the file backend, we can also report the size for raw images access
via the file backend.

Change-Id: I89118301c98dc2d11c25a4d1e7ef83df26336f01
Related: oVirt#67
Bug-Url: https://bugzilla.redhat.com/1924945
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
aesteve-rh pushed a commit to nirs/ovirt-imageio that referenced this issue Aug 8, 2022
Report image virtual size in OPTIONS so clients can get the image size
without a possibly slow extent call.

Report the format since OPTIONS can report the virtual size only when
the backend provides raw format. This is used when using the http
backend to report OPTIONS to the client.

Reporting virtual size is easy with the nbd and memory backends since
they always use raw format.

When using file backend and qcow2 image, we don't have access to the
image virtual size, and this size is not helpful to the user uploading
or downloading data. Currently we don't know about the image format
since engine does not report it in the ticket.

The http backend reports the info from the remote server, so it depends
on the backend used by the remote server, and on having new server
reporting the format and size.

To keep code and the API simple, we report virtual size only when using
the nbd and memory backends. When engine will report the image format
for the file backend, we can also report the size for raw images access
via the file backend.

Change-Id: I89118301c98dc2d11c25a4d1e7ef83df26336f01
Related: oVirt#67
Bug-Url: https://bugzilla.redhat.com/1924945
Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancing the system by adding new feature or improving performance or reliability performance
Projects
None yet
Development

No branches or pull requests

2 participants