1//! This is a densely packed error representation which is used on targets with
2//! 64-bit pointers.
3//!
4//! (Note that `bitpacked` vs `unpacked` here has no relationship to
5//! `#[repr(packed)]`, it just refers to attempting to use any available bits in
6//! a more clever manner than `rustc`'s default layout algorithm would).
7//!
8//! Conceptually, it stores the same data as the "unpacked" equivalent we use on
9//! other targets. Specifically, you can imagine it as an optimized version of
10//! the following enum (which is roughly equivalent to what's stored by
11//! `repr_unpacked::Repr`, e.g. `super::ErrorData<Box<Custom>>`):
12//!
13//! ```ignore (exposition-only)
14//! enum ErrorData {
15//! Os(i32),
16//! Simple(ErrorKind),
17//! SimpleMessage(&'static SimpleMessage),
18//! Custom(Box<Custom>),
19//! }
20//! ```
21//!
22//! However, it packs this data into a 64bit non-zero value.
23//!
24//! This optimization not only allows `io::Error` to occupy a single pointer,
25//! but improves `io::Result` as well, especially for situations like
26//! `io::Result<()>` (which is now 64 bits) or `io::Result<u64>` (which is now
27//! 128 bits), which are quite common.
28//!
29//! # Layout
30//! Tagged values are 64 bits, with the 2 least significant bits used for the
31//! tag. This means there are 4 "variants":
32//!
33//! - **Tag 0b00**: The first variant is equivalent to
34//! `ErrorData::SimpleMessage`, and holds a `&'static SimpleMessage` directly.
35//!
36//! `SimpleMessage` has an alignment >= 4 (which is requested with
37//! `#[repr(align)]` and checked statically at the bottom of this file), which
38//! means every `&'static SimpleMessage` should have the both tag bits as 0,
39//! meaning its tagged and untagged representation are equivalent.
40//!
41//! This means we can skip tagging it, which is necessary as this variant can
42//! be constructed from a `const fn`, which probably cannot tag pointers (or
43//! at least it would be difficult).
44//!
45//! - **Tag 0b01**: The other pointer variant holds the data for
46//! `ErrorData::Custom` and the remaining 62 bits are used to store a
47//! `Box<Custom>`. `Custom` also has alignment >= 4, so the bottom two bits
48//! are free to use for the tag.
49//!
50//! The only important thing to note is that `ptr::wrapping_add` and
51//! `ptr::wrapping_sub` are used to tag the pointer, rather than bitwise
52//! operations. This should preserve the pointer's provenance, which would
53//! otherwise be lost.
54//!
55//! - **Tag 0b10**: Holds the data for `ErrorData::Os(i32)`. We store the `i32`
56//! in the pointer's most significant 32 bits, and don't use the bits `2..32`
57//! for anything. Using the top 32 bits is just to let us easily recover the
58//! `i32` code with the correct sign.
59//!
60//! - **Tag 0b11**: Holds the data for `ErrorData::Simple(ErrorKind)`. This
61//! stores the `ErrorKind` in the top 32 bits as well, although it doesn't
62//! occupy nearly that many. Most of the bits are unused here, but it's not
63//! like we need them for anything else yet.
64//!
65//! # Use of `NonNull<()>`
66//!
67//! Everything is stored in a `NonNull<()>`, which is odd, but actually serves a
68//! purpose.
69//!
70//! Conceptually you might think of this more like:
71//!
72//! ```ignore (exposition-only)
73//! union Repr {
74//! // holds integer (Simple/Os) variants, and
75//! // provides access to the tag bits.
76//! bits: NonZero<u64>,
77//! // Tag is 0, so this is stored untagged.
78//! msg: &'static SimpleMessage,
79//! // Tagged (offset) `Box<Custom>` pointer.
80//! tagged_custom: NonNull<()>,
81//! }
82//! ```
83//!
84//! But there are a few problems with this:
85//!
86//! 1. Union access is equivalent to a transmute, so this representation would
87//! require we transmute between integers and pointers in at least one
88//! direction, which may be UB (and even if not, it is likely harder for a
89//! compiler to reason about than explicit ptr->int operations).
90//!
91//! 2. Even if all fields of a union have a niche, the union itself doesn't,
92//! although this may change in the future. This would make things like
93//! `io::Result<()>` and `io::Result<usize>` larger, which defeats part of
94//! the motivation of this bitpacking.
95//!
96//! Storing everything in a `NonZero<usize>` (or some other integer) would be a
97//! bit more traditional for pointer tagging, but it would lose provenance
98//! information, couldn't be constructed from a `const fn`, and would probably
99//! run into other issues as well.
100//!
101//! The `NonNull<()>` seems like the only alternative, even if it's fairly odd
102//! to use a pointer type to store something that may hold an integer, some of
103//! the time.
104
105use core::marker::PhantomData;
106use core::num::NonZeroUsize;
107use core::ptr::NonNull;
108
109use super::{Custom, ErrorData, ErrorKind, RawOsError, SimpleMessage};
110
111// The 2 least-significant bits are used as tag.
112const TAG_MASK: usize = 0b11;
113const TAG_SIMPLE_MESSAGE: usize = 0b00;
114const TAG_CUSTOM: usize = 0b01;
115const TAG_OS: usize = 0b10;
116const TAG_SIMPLE: usize = 0b11;
117
118/// The internal representation.
119///
120/// See the module docs for more, this is just a way to hack in a check that we
121/// indeed are not unwind-safe.
122///
123/// ```compile_fail,E0277
124/// fn is_unwind_safe<T: core::panic::UnwindSafe>() {}
125/// is_unwind_safe::<std::io::Error>();
126/// ```
127#[repr(transparent)]
128#[rustc_insignificant_dtor]
129pub(super) struct Repr(NonNull<()>, PhantomData<ErrorData<Box<Custom>>>);
130
131// All the types `Repr` stores internally are Send + Sync, and so is it.
132unsafe impl Send for Repr {}
133unsafe impl Sync for Repr {}
134
135impl Repr {
136 pub(super) fn new(dat: ErrorData<Box<Custom>>) -> Self {
137 match dat {
138 ErrorData::Os(code) => Self::new_os(code),
139 ErrorData::Simple(kind) => Self::new_simple(kind),
140 ErrorData::SimpleMessage(simple_message) => Self::new_simple_message(simple_message),
141 ErrorData::Custom(b) => Self::new_custom(b),
142 }
143 }
144
145 pub(super) fn new_custom(b: Box<Custom>) -> Self {
146 let p = Box::into_raw(b).cast::<u8>();
147 // Should only be possible if an allocator handed out a pointer with
148 // wrong alignment.
149 debug_assert_eq!(p.addr() & TAG_MASK, 0);
150 // Note: We know `TAG_CUSTOM <= size_of::<Custom>()` (static_assert at
151 // end of file), and both the start and end of the expression must be
152 // valid without address space wraparound due to `Box`'s semantics.
153 //
154 // This means it would be correct to implement this using `ptr::add`
155 // (rather than `ptr::wrapping_add`), but it's unclear this would give
156 // any benefit, so we just use `wrapping_add` instead.
157 let tagged = p.wrapping_add(TAG_CUSTOM).cast::<()>();
158 // Safety: `TAG_CUSTOM + p` is the same as `TAG_CUSTOM | p`,
159 // because `p`'s alignment means it isn't allowed to have any of the
160 // `TAG_BITS` set (you can verify that addition and bitwise-or are the
161 // same when the operands have no bits in common using a truth table).
162 //
163 // Then, `TAG_CUSTOM | p` is not zero, as that would require
164 // `TAG_CUSTOM` and `p` both be zero, and neither is (as `p` came from a
165 // box, and `TAG_CUSTOM` just... isn't zero -- it's `0b01`). Therefore,
166 // `TAG_CUSTOM + p` isn't zero and so `tagged` can't be, and the
167 // `new_unchecked` is safe.
168 let res = Self(unsafe { NonNull::new_unchecked(tagged) }, PhantomData);
169 // quickly smoke-check we encoded the right thing (This generally will
170 // only run in std's tests, unless the user uses -Zbuild-std)
171 debug_assert!(matches!(res.data(), ErrorData::Custom(_)), "repr(custom) encoding failed");
172 res
173 }
174
175 #[inline]
176 pub(super) fn new_os(code: RawOsError) -> Self {
177 let utagged = ((code as usize) << 32) | TAG_OS;
178 // Safety: `TAG_OS` is not zero, so the result of the `|` is not 0.
179 let res = Self(
180 NonNull::without_provenance(unsafe { NonZeroUsize::new_unchecked(utagged) }),
181 PhantomData,
182 );
183 // quickly smoke-check we encoded the right thing (This generally will
184 // only run in std's tests, unless the user uses -Zbuild-std)
185 debug_assert!(
186 matches!(res.data(), ErrorData::Os(c) if c == code),
187 "repr(os) encoding failed for {code}"
188 );
189 res
190 }
191
192 #[inline]
193 pub(super) fn new_simple(kind: ErrorKind) -> Self {
194 let utagged = ((kind as usize) << 32) | TAG_SIMPLE;
195 // Safety: `TAG_SIMPLE` is not zero, so the result of the `|` is not 0.
196 let res = Self(
197 NonNull::without_provenance(unsafe { NonZeroUsize::new_unchecked(utagged) }),
198 PhantomData,
199 );
200 // quickly smoke-check we encoded the right thing (This generally will
201 // only run in std's tests, unless the user uses -Zbuild-std)
202 debug_assert!(
203 matches!(res.data(), ErrorData::Simple(k) if k == kind),
204 "repr(simple) encoding failed {:?}",
205 kind,
206 );
207 res
208 }
209
210 #[inline]
211 pub(super) const fn new_simple_message(m: &'static SimpleMessage) -> Self {
212 // Safety: References are never null.
213 Self(unsafe { NonNull::new_unchecked(m as *const _ as *mut ()) }, PhantomData)
214 }
215
216 #[inline]
217 pub(super) fn data(&self) -> ErrorData<&Custom> {
218 // Safety: We're a Repr, decode_repr is fine.
219 unsafe { decode_repr(self.0, |c| &*c) }
220 }
221
222 #[inline]
223 pub(super) fn data_mut(&mut self) -> ErrorData<&mut Custom> {
224 // Safety: We're a Repr, decode_repr is fine.
225 unsafe { decode_repr(self.0, |c| &mut *c) }
226 }
227
228 #[inline]
229 pub(super) fn into_data(self) -> ErrorData<Box<Custom>> {
230 let this = core::mem::ManuallyDrop::new(self);
231 // Safety: We're a Repr, decode_repr is fine. The `Box::from_raw` is
232 // safe because we prevent double-drop using `ManuallyDrop`.
233 unsafe { decode_repr(this.0, |p| Box::from_raw(p)) }
234 }
235}
236
237impl Drop for Repr {
238 #[inline]
239 fn drop(&mut self) {
240 // Safety: We're a Repr, decode_repr is fine. The `Box::from_raw` is
241 // safe because we're being dropped.
242 unsafe {
243 let _ = decode_repr(self.0, |p: *mut Custom| Box::<Custom>::from_raw(p));
244 }
245 }
246}
247
248// Shared helper to decode a `Repr`'s internal pointer into an ErrorData.
249//
250// Safety: `ptr`'s bits should be encoded as described in the document at the
251// top (it should `some_repr.0`)
252#[inline]
253unsafe fn decode_repr<C, F>(ptr: NonNull<()>, make_custom: F) -> ErrorData<C>
254where
255 F: FnOnce(*mut Custom) -> C,
256{
257 let bits = ptr.as_ptr().addr();
258 match bits & TAG_MASK {
259 TAG_OS => {
260 let code = ((bits as i64) >> 32) as RawOsError;
261 ErrorData::Os(code)
262 }
263 TAG_SIMPLE => {
264 let kind_bits = (bits >> 32) as u32;
265 let kind = kind_from_prim(kind_bits).unwrap_or_else(|| {
266 debug_assert!(false, "Invalid io::error::Repr bits: `Repr({:#018x})`", bits);
267 // This means the `ptr` passed in was not valid, which violates
268 // the unsafe contract of `decode_repr`.
269 //
270 // Using this rather than unwrap meaningfully improves the code
271 // for callers which only care about one variant (usually
272 // `Custom`)
273 unsafe { core::hint::unreachable_unchecked() };
274 });
275 ErrorData::Simple(kind)
276 }
277 TAG_SIMPLE_MESSAGE => {
278 // SAFETY: per tag
279 unsafe { ErrorData::SimpleMessage(&*ptr.cast::<SimpleMessage>().as_ptr()) }
280 }
281 TAG_CUSTOM => {
282 // It would be correct for us to use `ptr::byte_sub` here (see the
283 // comment above the `wrapping_add` call in `new_custom` for why),
284 // but it isn't clear that it makes a difference, so we don't.
285 let custom = ptr.as_ptr().wrapping_byte_sub(TAG_CUSTOM).cast::<Custom>();
286 ErrorData::Custom(make_custom(custom))
287 }
288 _ => {
289 // Can't happen, and compiler can tell
290 unreachable!();
291 }
292 }
293}
294
295// This compiles to the same code as the check+transmute, but doesn't require
296// unsafe, or to hard-code max ErrorKind or its size in a way the compiler
297// couldn't verify.
298#[inline]
299fn kind_from_prim(ek: u32) -> Option<ErrorKind> {
300 macro_rules! from_prim {
301 ($prim:expr => $Enum:ident { $($Variant:ident),* $(,)? }) => {{
302 // Force a compile error if the list gets out of date.
303 const _: fn(e: $Enum) = |e: $Enum| match e {
304 $($Enum::$Variant => ()),*
305 };
306 match $prim {
307 $(v if v == ($Enum::$Variant as _) => Some($Enum::$Variant),)*
308 _ => None,
309 }
310 }}
311 }
312 from_prim!(ek => ErrorKind {
313 NotFound,
314 PermissionDenied,
315 ConnectionRefused,
316 ConnectionReset,
317 HostUnreachable,
318 NetworkUnreachable,
319 ConnectionAborted,
320 NotConnected,
321 AddrInUse,
322 AddrNotAvailable,
323 NetworkDown,
324 BrokenPipe,
325 AlreadyExists,
326 WouldBlock,
327 NotADirectory,
328 IsADirectory,
329 DirectoryNotEmpty,
330 ReadOnlyFilesystem,
331 FilesystemLoop,
332 StaleNetworkFileHandle,
333 InvalidInput,
334 InvalidData,
335 TimedOut,
336 WriteZero,
337 StorageFull,
338 NotSeekable,
339 QuotaExceeded,
340 FileTooLarge,
341 ResourceBusy,
342 ExecutableFileBusy,
343 Deadlock,
344 CrossesDevices,
345 TooManyLinks,
346 InvalidFilename,
347 ArgumentListTooLong,
348 Interrupted,
349 Other,
350 UnexpectedEof,
351 Unsupported,
352 OutOfMemory,
353 InProgress,
354 Uncategorized,
355 })
356}
357
358// Some static checking to alert us if a change breaks any of the assumptions
359// that our encoding relies on for correctness and soundness. (Some of these are
360// a bit overly thorough/cautious, admittedly)
361//
362// If any of these are hit on a platform that std supports, we should likely
363// just use `repr_unpacked.rs` there instead (unless the fix is easy).
364macro_rules! static_assert {
365 ($condition:expr) => {
366 const _: () = assert!($condition);
367 };
368 (@usize_eq: $lhs:expr, $rhs:expr) => {
369 const _: [(); $lhs] = [(); $rhs];
370 };
371}
372
373// The bitpacking we use requires pointers be exactly 64 bits.
374static_assert!(@usize_eq: size_of::<NonNull<()>>(), 8);
375
376// We also require pointers and usize be the same size.
377static_assert!(@usize_eq: size_of::<NonNull<()>>(), size_of::<usize>());
378
379// `Custom` and `SimpleMessage` need to be thin pointers.
380static_assert!(@usize_eq: size_of::<&'static SimpleMessage>(), 8);
381static_assert!(@usize_eq: size_of::<Box<Custom>>(), 8);
382
383static_assert!((TAG_MASK + 1).is_power_of_two());
384// And they must have sufficient alignment.
385static_assert!(align_of::<SimpleMessage>() >= TAG_MASK + 1);
386static_assert!(align_of::<Custom>() >= TAG_MASK + 1);
387
388static_assert!(@usize_eq: TAG_MASK & TAG_SIMPLE_MESSAGE, TAG_SIMPLE_MESSAGE);
389static_assert!(@usize_eq: TAG_MASK & TAG_CUSTOM, TAG_CUSTOM);
390static_assert!(@usize_eq: TAG_MASK & TAG_OS, TAG_OS);
391static_assert!(@usize_eq: TAG_MASK & TAG_SIMPLE, TAG_SIMPLE);
392
393// This is obviously true (`TAG_CUSTOM` is `0b01`), but in `Repr::new_custom` we
394// offset a pointer by this value, and expect it to both be within the same
395// object, and to not wrap around the address space. See the comment in that
396// function for further details.
397//
398// Actually, at the moment we use `ptr::wrapping_add`, not `ptr::add`, so this
399// check isn't needed for that one, although the assertion that we don't
400// actually wrap around in that wrapping_add does simplify the safety reasoning
401// elsewhere considerably.
402static_assert!(size_of::<Custom>() >= TAG_CUSTOM);
403
404// These two store a payload which is allowed to be zero, so they must be
405// non-zero to preserve the `NonNull`'s range invariant.
406static_assert!(TAG_OS != 0);
407static_assert!(TAG_SIMPLE != 0);
408// We can't tag `SimpleMessage`s, the tag must be 0.
409static_assert!(@usize_eq: TAG_SIMPLE_MESSAGE, 0);
410
411// Check that the point of all of this still holds.
412//
413// We'd check against `io::Error`, but *technically* it's allowed to vary,
414// as it's not `#[repr(transparent)]`/`#[repr(C)]`. We could add that, but
415// the `#[repr()]` would show up in rustdoc, which might be seen as a stable
416// commitment.
417static_assert!(@usize_eq: size_of::<Repr>(), 8);
418static_assert!(@usize_eq: size_of::<Option<Repr>>(), 8);
419static_assert!(@usize_eq: size_of::<Result<(), Repr>>(), 8);
420static_assert!(@usize_eq: size_of::<Result<usize, Repr>>(), 16);
421