zae_engine.nn_night.layers package

Submodules

zae_engine.nn_night.layers.additional module

class zae_engine.nn_night.layers.additional.Additional(*args)

기반 클래스: ModuleList

Additional connection module.

This module extends nn.ModuleList to implement an additional connection. Each input tensor is passed through its corresponding module, and the output tensors are summed. If the shapes of the output tensors do not match, an error is raised.

매개변수:

*args (nn.Module) – Sequence of PyTorch modules. Each module will be applied to a corresponding input tensor in the forward pass.

forward(*inputs)

Applies each module to its corresponding input tensor and returns the sum of the output tensors. If the shapes of the output tensors do not match, an error is raised.

forward(*inputs)

Forward pass through the additional block.

Applies each module to its corresponding input tensor and returns the sum of the output tensors. If the shapes of the output tensors do not match, an error is raised.

매개변수:

*inputs (torch.Tensor) – Sequence of input tensors. Each tensor is passed through its corresponding module.

반환:

The sum of the output tensors of each module.

반환 형식:

torch.Tensor

예외 발생:

ValueError – If the output tensors have mismatched shapes.

zae_engine.nn_night.layers.dynamic_pool module

class zae_engine.nn_night.layers.dynamic_pool.DynOPool

기반 클래스: Module

Dynamic Pooling Layer using Gumbel Softmax trick for discrete pooling ratios.

This layer dynamically adjusts the pooling ratio using a learnable parameter, allowing for adaptive pooling during training. The Gumbel Softmax trick is applied to ensure the ratio remains discrete.

Reference:

• DynOPool: Dynamic Optimization Pooling, https://arxiv.org/abs/2205.15254

ratio

Learnable parameter representing the pooling ratio.

Type:

nn.Parameter

trick

Function to apply the Gumbel Softmax trick.

Type:

function

bilinear_interpolation(x, to_dim)

Performs bilinear interpolation on the input tensor to the specified dimension.

forward(x)

Forward pass for the DynOPool layer.

bilinear_interpolation(x, to_dim)

Perform bilinear interpolation on the input tensor to the specified dimension.

매개변수:

• x (torch.Tensor) – Input tensor of shape (batch_size, channels, depth).

• to_dim (torch.Tensor) – Target dimension after interpolation.

반환:

Interpolated tensor.

반환 형식:

torch.Tensor

forward(x)

Forward pass for the DynOPool layer.

매개변수:

x (torch.Tensor) – Input tensor of shape (batch_size, channels, depth).

반환:

Pooled tensor.

반환 형식:

torch.Tensor

zae_engine.nn_night.layers.involution module

class zae_engine.nn_night.layers.involution.Inv1d(ch: int, num_groups: int, kernel_size: int, stride: int, reduction_ratio: int)

기반 클래스: Module

The involution layer for 1D input [1]_.

… :param ch: Number of channels in the input tensor (signal or 1D arr) :type ch: int :param num_groups: Number of channels produced by the convolution :type num_groups: int :param kernel_size: Size of the convolving kernel :type kernel_size: int :param stride: Stride of the convolution. Default: 1 :type stride: int :param reduction_ratio: Ratio of channel reduction. This value must be divisor of ch. :type reduction_ratio: int

반환:

tensor

반환 형식:

tensor

참조

[1]

Li, Duo, et al. “Involution: Inverting the inherence of convolution for visual recognition.” Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

forward(x)

Defines the computation performed at every call.

Should be overridden by all subclasses.

참고

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

kernel_generator()

zae_engine.nn_night.layers.position_encoding module

class zae_engine.nn_night.layers.position_encoding.AdaptivePositionalEncoding(d_model, max_len=512)

기반 클래스: Module

Implements adaptive positional encoding that adjusts position encoding based on input sequence length.

매개변수:

• d_model (int) – Dimension of the embedding space.

• max_len (int, optional) – Maximum sequence length. Default is 512.

참고

• This method adjusts the position encoding dynamically based on sequence length.

• Benefits: Flexible for handling sequences of varying lengths.

• Drawbacks: Requires additional handling for sequences with different lengths.

참조

• No specific reference, this approach is inspired by the need for adaptive positional encodings in models handling variable-length sequences.

forward(x, **kwargs)

Apply adaptive positional encoding to input tensor based on sequence length.

매개변수:

• x (torch.Tensor) – Input tensor of shape (batch_size, seq_len, d_model).

• **kwargs (dict) – Additional keyword arguments, such as ‘seq_lengths’ for handling variable-length sequences.

반환:

Tensor with added positional encoding.

반환 형식:

torch.Tensor

class zae_engine.nn_night.layers.position_encoding.LearnablePositionalEncoding(d_model, max_len=512)

기반 클래스: Module

Implements learnable positional encoding where positional embeddings are learned during training.

매개변수:

• d_model (int) – Dimension of the embedding space.

• max_len (int, optional) – Maximum sequence length. Default is 512.

참고

• This method allows the model to learn optimal positional encodings during training.

• Benefits: Can adapt the positional encoding to the specific task.

• Drawbacks: Requires additional parameters and training time.

참조

• No specific reference, this approach is commonly used in various models including BERT and GPT.

forward(x)

Apply learnable positional encoding to input tensor.

매개변수:

x (torch.Tensor) – Input tensor of shape (batch_size, seq_len, d_model).

반환:

Tensor with added positional encoding.

반환 형식:

torch.Tensor

class zae_engine.nn_night.layers.position_encoding.RelativePositionalEncoding(d_model, max_len=512)

기반 클래스: Module

Implements relative positional encoding that captures relative distances between tokens.

매개변수:

• d_model (int) – Dimension of the embedding space.

• max_len (int, optional) – Maximum sequence length. Default is 512.

참고

• This method is used in models like Transformer-XL and T5 [2]_[3]_.

• Benefits: Handles long sequences and captures relative positions.

• Drawbacks: May increase computational complexity.

참조

[2]

Dai, Z., Yang, Z., Yang, Y., Carbonell, J. G., Salakhutdinov, R., & Liu, T.-Y. (2019). Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019). Available at: https://arxiv.org/abs/1901.02860 DOI: 10.48550/arXiv.1901.02860

[3]

Raffel, C., Shinn, C., Gauthier, J., & others (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21, 1-67. Available at: https://arxiv.org/abs/1910.10683 DOI: 10.48550/arXiv.1910.10683

forward(x)

Apply relative positional encoding to input tensor.

매개변수:

x (torch.Tensor) – Input tensor of shape (batch_size, seq_len, d_model).

반환:

Tensor with added positional encoding.

반환 형식:

torch.Tensor

class zae_engine.nn_night.layers.position_encoding.RotaryPositionalEncoding(d_model)

기반 클래스: Module

Implements Rotary Positional Encoding as described in “RoFormer: Enhanced Transformer with Rotary Position Embedding”.

매개변수:

d_model (int) – Dimension of the embedding space. Must be divisible by 2 for rotary encoding.

forward(x)

Apply rotary positional encoding to input tensor.

매개변수:

x (torch.Tensor) – Input tensor of shape (batch_size, seq_len, d_model).

반환:

Tensor with added rotary positional encoding.

반환 형식:

torch.Tensor

class zae_engine.nn_night.layers.position_encoding.SinusoidalPositionalEncoding(d_model, max_len=512)

기반 클래스: Module

Computes sinusoidal positional encoding as described in the Transformer paper [1]_.

매개변수:

• d_model (int) – Dimension of the embedding space. Must be an even number.

• max_len (int, optional) – Maximum sequence length. Default is 512.

참고

• This method was introduced in the original Transformer paper (Vaswani et al., 2017) [1]_.

• Uses fixed sine and cosine functions of different frequencies to encode token positions.

• Benefits: Simple and efficient to compute, and captures positional information effectively.

• Drawbacks: Does not capture relative positional information.

참조

[1]

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.Õ., Kaiser, Ł., Polosukhin, I. (2017). Attention is All You Need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS 2017). Available at: https://arxiv.org/abs/1706.03762 DOI: 10.48550/arXiv.1706.03762

forward(x, positions: Tensor = None)

Apply sinusoidal positional encoding to the input tensor.

매개변수:

• x (torch.Tensor) – Input tensor of shape (batch_size, seq_len, d_model).

• positions (torch.Tensor, optional) – Optional tensor of shape (batch_size, seq_len) specifying the positions (e.g., timestamps) for each element in the sequence. If not provided, the default positions (0 to seq_len - 1) are used.

반환:

Tensor with added positional encoding.

반환 형식:

torch.Tensor

zae_engine.nn_night.layers.residual_connection module

class zae_engine.nn_night.layers.residual_connection.Residual(*args)

기반 클래스: Sequential

Residual connection module.

This module extends nn.Sequential to implement a residual connection. The input tensor is added to the output tensor of the sequence of modules provided during initialization, similar to a residual block in ResNet architectures.

매개변수:

*args (nn.Module) – Sequence of PyTorch modules to be applied to the input tensor.

forward(x)

Applies the sequence of modules to the input tensor and returns the sum of the input tensor and the output tensor.

forward(x)

Forward pass through the residual block.

Applies the sequence of modules to the input tensor and returns the sum of the input tensor and the output tensor.

매개변수:

x (torch.Tensor) – Input tensor.

반환:

The sum of the input tensor and the output of the sequence of modules.

반환 형식:

torch.Tensor

zae_engine.nn_night.layers.selective_kernel_conv module

class zae_engine.nn_night.layers.selective_kernel_conv.SKConv1D(ch_in: int, ch_out: int = None, kernels: list = (3, 5), out_size: int = None, ch_ratio: int = 2, stride: int = 1)

기반 클래스: Module

Selective Kernel Convolution for 1D inputs.

This module performs selective kernel convolutions with different kernel sizes, followed by a selection mechanism to fuse the features.

매개변수:

• ch_in (int) – Number of input channels.

• ch_out (int, optional) – Number of output channels. If None, it will be set to the same as ch_in.

• kernels (list or tuple, optional) – List of kernel sizes to be used in the convolution layers. Default is (3, 5).

• out_size (int, optional) – Output size for adaptive average pooling. If None, no pooling is applied.

• ch_ratio (int, optional) – Reduction ratio for the intermediate channels in the fuse layer. Default is 2.

• stride (int, optional) – Stride size for the convolution layers. Default is 1.

참조

[1]

Li, X., Wang, W., Hu, X., & Yang, J. (2019). Selective kernel networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 510-519). Available at: https://arxiv.org/abs/1903.06586

forward(x: Tensor) → Tensor

Defines the computation performed at every call.

Should be overridden by all subclasses.

참고

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

fuse(ch_in: int, ch_out: int) → Sequential

kernel_valid()

Module contents