Empresas
Empregos
  • Sobre nós
  • Soluções
    • Publicação de vagas
      Publique sua vaga e receba candidatos qualificados em 48h.
    • Avaliações de candidatos
      Mais de 500 testes técnicos e psicológicos, mais anti-fraude.
    • Headhunting
      Busca executiva personalizada do início ao fim.
    • Folha de Pagamento + EOR
      Dispersão de folha e EOR em mais de 15 países da LATAM.
  • Preços
  • Empregos

0

433
Visualizações
Is there a way to stack two tensorflow datasets?

I want to stack two datasets objects in Tensorflow (rbind function in R). I have created one dataset A from tfRecord files and one dataset B from numpy arrays. Both have same variables. Do you know if there is a way to stack these two datasets to create a bigger one ? Or to create an iterrator that will randomly read data from this two sources ?

Thanks

over 4 years ago · Santiago Trujillo
3 Respostas
Responde à pergunta

0

The tf.data.Dataset.concatenate() method is the closest analog of tf.stack() when working with datasets. If you have two datasets with the same structure (i.e. same types for each component, but possibly different shapes):

dataset_1 = tf.data.Dataset.range(10, 20)
dataset_2 = tf.data.Dataset.range(60, 70)

then you can concatenate them as follows:

combined_dataset = dataset_1.concatenate(dataset_2)
over 4 years ago · Santiago Trujillo Relatório

0

If by stacking you mean what tf.stack() and np.stack() do:

Stacks a list of rank-R tensors into one rank-(R+1) tensor.

https://www.tensorflow.org/api_docs/python/tf/stack

Join a sequence of arrays along a new axis.

https://docs.scipy.org/doc/numpy/reference/generated/numpy.stack.html

then I believe the closest you can come with a tf.data.Dataset is Dataset.zip():

@staticmethod
zip(datasets)

Creates a Dataset by zipping together the given datasets.

https://www.tensorflow.org/api_docs/python/tf/data/Dataset?version=stable#zip

This allows you to iterate through multiple datasets at the same time by iterating over the shared dimension of the original datasets, similarly to a stack()ed tensor or matrix.

You can then also use .map(tf.stack) or .map(lambda *t: tf.stack(t, axis=-1)) to stack the tensors along new dimensions at the front or back, respectively,

If indeed you want to achieve what tf.concat() and np.concatenate() do, then you use Dataset.concatenate().

over 4 years ago · Santiago Trujillo Relatório

0

Assume you have two datasets which elements shape is respectively (bs,d0,d1) and (bs,d0',d1) and you want a new dataset which element shape is (bs,d0+d0',d1) you can do it using tf.Dataset.zip and then concatenating each element on the second axis, like in the example below:

import tensorflow as tf

a = tf.zeros((100,4,8))
b = tf.ones((100,1,8))

d1 = tf.data.Dataset.from_tensor_slices(a)
d1 = d1.batch(16,drop_remainder=True)      # elements shape (16,4,8)

d2 = tf.data.Dataset.from_tensor_slices(b)
d2 = d2.batch(16,drop_remainder=True)      # elements shape (16,1,8)

d = tf.data.Dataset.zip((d1,d2))
d = d.map(lambda x,y:tf.concat([x,y],axis=-2)) # elements shape (16,4+1,8)

it = iter(d)
x = next(it)
print(x.shape)
print(x)

If you want instead to stack two datasets with the same elements shape (bs,d0,d1) into a new dataset with elements shape (bs,d0,d1,2) you can do it zipping the two datasets and then staking the elements

import tensorflow as tf

a = tf.zeros((100,4,8))
b = tf.ones((100,4,8))

d1 = tf.data.Dataset.from_tensor_slices(a)
d1 = d1.batch(16,drop_remainder=True)      # elements shape (16,4,8)

d2 = tf.data.Dataset.from_tensor_slices(b)
d2 = d2.batch(16,drop_remainder=True)      # elements shape (16,4,8)

d = tf.data.Dataset.zip((d1,d2))
d = d.map(lambda x,y:tf.stack([x,y],axis=-1)) # elements shape (16,4,8,2)

it = iter(d)
x = next(it)
print(x.shape)
print(x)
over 4 years ago · Santiago Trujillo Relatório
Responde à pergunta
Encontrar trabalhos remotos

Descubra a nova forma de encontrar um emprego!

melhores empregos
Principais categorias de trabalho
Empresas
Postar vaga Preços Comercial
Jurídico
Termos e Condições Política de privacidade
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomende algumas ofertas para mim
Preciso de ajuda